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ABSTRACT / 

A progress re^rt describes the desigi; and 
demonstration of the compute-r- based Instructional Management Systems 
(IMS) for the period March 1 , 1967, through February 29, 19 68 . The. 
rationale and design considerations of IMS, development of its system 
components, functioning of the components during the first field 
trial, and data collected during the first field trial are discussed. 
In IMS, teachers administer frequent tests, which are keyed to 
behaviorally defined objectives and print^ on machine readable 
forms. Data... input procedures transform pupil responses for computer 
analysis, and the computer-generated report contains an analysis of 
pupils' achievement of objectives and suggests remedial exercises for 
those who do not meet the criterion performance level. (JY) 
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ABSTRACT 



This document describes progress in the development and demonstration of the ___ 
computer-based Instructional Management System (IMS) during. thTpeHod MarSTT,” 
1967 ^ through February 29^ 1968. The rationale and desi^ considerations of IMS, 
development of its system components, functioning of the components during the 
first field trial, and data collected during the first field trial are discussed 
in this progress report. 

BIS is designed to aid teachers in the individual monitoring and management 
of student progress. The teachers administer frequent tests, which are keyed 
to behaviorally defined objectives and printed on machine readable forma. Data 
input procedures transform pupil responses for computer analysis, and the computer 
generates a report to be delivered to the teacher the next day. The con5>uter- 
generated rei>ort contains an analysis of pupils* achievement of objectives and 
suggests remedial exercises for those who do not meet the criterion performance 
level. ■ . : • 

The initial field trial of IMS began in September, I967, with four first-grade 
classes from two Los Angeles elementary schools participating. After a few 
trials and’ procedural adjustments, mechanical components of the system performed 
to specifications set for the first demonstration. However, analysis of available 
data revealed that teachers did not consistently use information provided by BIS 
to make instructional decisions. New materials and procedures were designed to 
improve the system for the spring I968 school term. 
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The basic premise underlying the design of INB is that classroom instruction 
can be made more individu^ized and effective only if the school has accurate. 

rapidly the students are progressing, (2) what 
learning difficulties they are experiencing, and (3) what instructional materials 
or other resources are available to the school to remedy specific deficiencies ^ 
In student performance. 

Such information is clearly not available today in most schools. Except at the 
1° I intuitive level, most teachers have minimal knowledge about the class 

o a whole, let alone the specific strengths and weaknesses of individual pupils. 
Tests are so infrequently administered that they offer little day-to-day guidance 
to the teacher. Moreover, these tests are seldom designed to diagnose specific 
rning deficiencies, but instead yield gross\ measures of overall performance. 

to provide normative ratings,', not to tell the teaser whether 
a given Pupil or group of pupils has successivel'y mastered a specific skill such 

to thl^upUB'See^ “ Direct the Instruction 

Seatwork exercises and homework assignments are often used by teachers to elicit 
"eMom^ pupil participation, but again, the results of these activitief are 
seldom used to guide the teacher in any specific fashion. Unless the teacher 
exceptionally light teaching,. load, or is unusually favored in clerical 

less^^S^^P r loo^ over every pupil response. He has e'/en 

-p? ? to analyze response patterns in any systematic way so as to identify 

^cific learning deficiencies. By the time he could perform such'an analvsis 

SfSr“t”^op?c instruction and on some entire:, 

^Sou?iis'aJailabirto Il“ incoi^lete information about the instructional 

material ^ often not aware, for example, of supplementary 

materi^ in the scnool or in the district office. More importantly he has no ^ 

comprehensive catalog of materials keyed to specific learninrSiectivL LT 
cannot effectively match resources to needs. learning objectives, so he 

^Ive-^eachS^^^+h"’^"'®^^ information deficiencies described above, it 
t 4.1^1 diagnostic and prescriptive information about students 

i^fo^LtioraSV stores pupil performance data and 

about available seatwork materials. As the pupils complete ea.-h 
instructional unit, they are given criterion-referenced diagnosU^tesL on 
machine -readable answer sheets. Test results are used to u^aS ^h^n^nir 
porfdrmance records, and on the day after each test the teacher receive^ 

instructional materials that the JeLhh 

computer s data base tor additional perfornance information. 
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B. IfC DESIGN CONSIDERATIONS • I 

Three major guidelines are applied in the design of ItC. The first- is that 
II-B should be a practical system for operational use in public schools; the 
second io that it should have the flexibility for evolutionary development; 
and the third is that it should incorporate a built-in self-correcting 
mechanism for successive improvement through empirical trials aid revisions. 
These guidelines are discussed below. ' 

Practicality. Practicality means that I^B must be user-oriented and that it 
must be cost-feasible . To insure a user-orientation, IMS was designed from 
its initial conception with the school teachers in mind. Teacriers and school 
^inistrators participated in the original planning and system design, and 
teachers aided in the development of materials and procedures used in INB. 

The fomats of printouts and other displays produced by IMS have been raodified 
several times to make them more easily understood and acted on by the teachers 
New functions are currently being added to IMS at the suggestion of teachers 
who have gained familiarity with the system and envision further capabilities 
that can aid them. 



Cost- feasibility is a major advantage stemming from the fact that IMS does 
not use the computer to interact directly with students. Few terminals are 
required by IMS, and the computer can operate largely in a batch- processing 
mode without waiting for inputs from humans operating at a much slower pace. 
For these reasons, IMS is much less expensive than CAl for an equal number of 
stutents. Once INB has progressed from its present developmental stage to a 

routine operating system, it should be capable of being supported directly by 
school districts. ° rr j 



Evolutionary Approach. An educational innovation may be sound in concept and 
design, yet it may encounter strong resistance in the schools if it requires 
any abrupt changes in the routine organization and management of school oper- 
„ations. for this reason, it is often more effective to take an evolutionary 
' steorso +h innovation is introduced through a succession of small 

S tte eS * personnel become well adjusted to each step before moving 



of is based on this evolutionary approach. The 

conventional public Lhools with 

orates most school's organization or structure. IMS presently 

oSanl^ationL lie I group-paced mode, because this is the most common 

it user^irShnn? . o '^here INB has been introduced. Furthermore, 

It uses the schools existing curricula, rather than attempting to force a new 

warireiilsw*%hr°°^^\ Instruction is provided by the teachers just as it 
hL Scisri^' T teachers in classrooms using I^B now 

source materials available to help the pupils reach specific leSninrobjec^i^^ 

e?rec?y?:oi ! designed initially to help teachers do their present job mL ' 
efiecti.ely, not to make drastic changes in the nature of that^Job. 
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Nevertheless, the long-range objective of IMS is to facilitate change in 

■ individualized mode of instruction, and I^B is designed 
_ with sufficient flexibility to operate in almost any instructional mode, Som ^ 

■ ^ lockstep system to a continuous-progress system in which every 

student progressed independently. It is anticipated that, as teachers beciie 

Jith the initial version of im, they will see more clearly the need 
• and oportunity for a higher degree of individualization and will request more 

IT direction. Thus? subsequent 

4 tte 3Sop?p=°So“eJ? the needsjebelved 

I- ! 

Experience in the developm^t of educational systems 

trLd out Sd r2) thL"n operate-effectively when it is first 

triea out, and (2) that no systematic improvement's vill take place in the RVRtem 

uraess there is a built-in improvement mechanism in the origiSj ^y ?em1e S! 

^ incorporates such a mechanism, becaug^n intrinsic par? of its operations 
in Which reflecting.thi^rformance of I^6 and of the school system 

t^nulh\l^ functions. Classroom performance reports and other displays available 
through IJ-B provide convejitent means for assessing the degree to which INB is 
serving its function as^an aid- to teachers in the monitoring and man^em^t of ' 

^ application of thlB self-oorrertln^ aS^lrSaf 

al.xady led to a number of modifications in the IMS design. 

II. GENERAL DESCRIPTION OF I NB 

Figure 1 is a simplified diagram of . the information flow in the initial tkk 
T he large boxj^ the left of the figure represents one of thna^^ictStfnl* 
schools, and the right-hand box represents the data- processing facility at^DC 

aL r,' ^ aa^aots teat sheets £^1. tSe teLtera 

Md brings tlM to Spc, where they are read by an optical scanner. ThrS^er 
Sudents representing, in digital form, the resfonses o? tte 

an “ “a8**la tape, which in turn is read by 

asBMlatfw??!!^ located at SDC. In the Q-32, the raw response data are . 
associated with an answer key to produce individual and group Kores on rrpy.o.,+ 
learning objectives The objectives are also associated iHo^^r 
specific seginents of seat work exercise materials Thus the nrintnn-fc a a 

hti^tideLt ate btt““to ' 

letrnttt ttn T “ Sroup of stutents tLt 

learning deficiency. These printouts are carried back to the. school anfl^in +v,o 

milbS ^^°f summary form for his pupils in his ' 

^ilbox. xf he wishes more detailed information about a particular student 

Thl resS?s o? background data such as 

rmp°u^^? 
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Figure 1 



IMS Information Flow (initial Design) 
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■' ni. I 4 S DEVELOPMENT PLAN ^ ' 

This report is primarily concerned with the contract period of March 1967 
.aiough teoruMy lyGd. However, to place the activities of this period in 
spective, Figure 2, below, shows the longer range .INE development plan 
cover.Lng the period from June I966 through February I969. 

From June 1966 tfirough December I966, the preliminary design of IMS was ' 

specification- Of "equipLnt rJ^i^^Lnts, 

programs, and the initiation of meetings with district 

"eie^t!oro%« 1®" City Schools tcvestablish criteria for 

'■ 4 a!ief in ,hi schools. A report of these activities is 

...stained m I ne Instructional Technology Module Progress Report dated 

'k in SDG-. document TM(L)-3298/ooi/ 00, "Progrfess Report: 

I....tn..-tional Technology Module," January 16, I9.67. , ^ 

' ■» 1 

September 1967, two Il-B demonstration 

•o'ectid thf r J Brentwood Elementary School) were 

^ initial package of diagnostic tests, and othei' LVB materials 

find tested^-n programs to provide HE data processing' were written 

. P, . s*':‘^ent carrels and associated listening posts were 

■■^trat- ' ^ schools, and’ four teachers ! from the demon- 

' extensive indoctrination and training in the use 

If wh!crt?^2*?rwS'«' version of BE became operational in September 1967, 

I'rale) in each nr th»*i 1 " classes (first semester of the first 

in , fL ^ demonstration schools. Operation! continued, in the 

fication° ff® semester, during which' time several modi- 

'ITtll demoSsi^SjS c^asL‘s?“°”" introduced In the ispring 1963 semester 

operation of the Q-32 version of HE during 
-.le o.). iug l^od tera, and procedures and materials will be further mndif-iPri ^ 

^°="^®"t®d“nfuser^*^^ 

s bf^rtri— r— 

thic- '•4- K fi i It-B to a form operable on an IBM 360/69 computer Tn 

^ should be possible to adapt the prop’rams eaRilv 4-n \ 

SVHiL will pro’oably acquire in the first hai? nr^oGc? ^^4- computer 

ihul SvJiL iJill then use it^ noLnL. ! W. It is anticipated 

selected sOioils !n ^he Jegio"" IMS services to 
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Also in the coming contract year, IMS project members will design and begin to 
develop major additions and modifications to the initial IKE, to enhance its 
capabilities. In particular, the prescriptive capabilities will be strengthened 
so t^t IMS can give teachers more active assistance in the design and preparation 
of exercise materials to remedy learning deficiencies detected by the system. 
Beyond the time span represented by Figure 2 (i.e., after March I969), the new 
prescriptive capabilities will be further developed until they are ready to be 
incorporated as a regular part of the IMS operations. 



IV. INITIAL I^E DESIGN 



A. TESTS AND PRESCRIPTIVE MATERIALS 



Leve ls of Testing . The Instructional Management System operates through a 
set of computer programs and clascroom testing procedures. Since tests have 
several functions in classroom management, I^B was designed to allow several 
levels of testing. To illustrate, the teacher may only wish to have tests 
scored and scores accumulated for each pupil over the semester; for this 
she would use the assessment level of BE. Or, she may wish to have pupil 
performance analyzed in relation to instructional objectives; in this case, 
she would employ the diagnostic level. If, in addition, she wanted to have 
the system suggest activities for the child who was not meeting particular 
objectives, she would use the prescriptive level of IMS. 

Each level of the system provides greater management capability but requires 
more teacher input for operation. In any level, pupil responses must be 
acceptable to the computer for analysis. To meet this requirement, INB pupils 
mark their answers on sheets that can be read' by an IBM optical scanner. 
Responses on these sheets can be scored and added to a 'data base of information 
within the computer. The assessment level of IMS is made up of such scoring 
and accumulating of scores. 



If the teacher wishes to use the diagnostic level, to find how a total test 
score breaks down into a pattern of skills, she provides a list of objectives 
and keys particular test items to those objectives. The objectives may be 
general, as 'The child can use map- reading aids provided in a legend," or thev 
may be specific, as "The child can indicate the distance between two points on 
a map that shows a scale of miles." In addition to reporting a pupil's overall 
score, the computer can then supply partial scores associated with objectives 
From these a teacher might learn, for example, that a child's failure was 
associated entirely with not recognizing irregularly spelled words, while he 
could decode regular words quite adequately. 

The third and most intensive use of the system, the prescriptive level, provides 
suggested lesson plans for the teacher as well as assessment and diagnostic infor- 
m tion.* For this use, lesson plans and materials associated -with each objective 
are provided. Then the computer can select procedures for teaching based on 
information gained about the performance of individuals and groups!' This infor- 
mation rs provided on a computer printout. SDC used the prefcriptive level of 
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em, with all of its objectives, tests, item keying, and suggested 

for the initial demonstration. Four first-grade 

cTastrooms participated and were provided the full semester's materials! 

Other teachers in the same schools were also encoureiged to use the system, 
and several of these teachers did use it at the assessment or diagnostic 
level. 

General Plan for Test Construction . Few teachers in conventional classes 
prepare a comprehensive set of tests covering an entire semester's work in 
advance of the semester. Typically, although a teacher may have a good 
overview of the total semester's course, she writes each individual test just 
before she administers it. This ad hoc approach to the preparation of tests 
was not practical for the initial IMS demonstration, partly because a completed 
set of mater i^s and procedures was needed in advance to guide the programmers 
in aexining the requisite computer programs. Consequently, a group effort was 
initiated, m^y months before the start of the fall. 1967 school term, to write 
IIB tests and select remedial materials for the 1967-06 school year. The 
members of the group included a team of teachers from two. Los Angeles schools, 

St ff members of the Southwest Regional Laboratory, and personnel from System 
Development Corporation. Details of the test development effort, which was 

one of the largest and most important in the total if.B project, are described 
in Appendix A. 

There was no attempt to develop a new or "ideal" instructional program. 

Instead project personnel tried to reflect faithfully the teachers' existing 
instructional objectives and sequence in a program of frequent tests. There 
wre two reasons for modeling materials on currently operating programs. 

First, there vra.s inadequate time to construct a new program or to adapt one 
to the ongoing, operations of the cooperating classrooms. Second, it was felt 

system reSur- 

^ achieve their own objectives than if it imposed a new- s6t 

Of objectives. . . 

Structure . A set of 50 prototype tests was developed to 
nn ^ S q ! °^rational system during the 1967-68 school year. This set 
-Thf readlL W,"" reading readiness, 27 in reading, and I7 in arithmetic. 

a^?+w?J r r ff the Ginn Basic Reading Series and the 

arittoetic tests follow the S.R. A. -Greater Cleveland materials, since thZe 

e the programs used in the Los Angeles schools where trials were conducted. 

The structure of tests in all three areeis is essentially similar Pnr +n-ie 

arS°of fSdi^r extensive set of tests, those in the 

area ot reading, will be described, 

A reading test was provided after every third story in the reader- on thP 
average this corresponded roughly to one week of cL sroom "st ruction Each 

test contained six or seven pages, constructed in a variety "rSma^s' Onf 
orma is illustrated in the phonic analysis sheet taken from test 16 (see 
Figure 3). The Iteee in this sheet require the pupil to S t^ etLilus 



R16 





CD CZD 



CD 




Figure 3* Test Sheet Exanple. (in the actual forms, shadings 
and answer spaces are printed in red. ) 
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word on the left and then mark the appropriate response picture on the right. 

The name of the correct response picture begins with the same letter as the 
stimulus word. Other formats used in the reading tests require matching a 
word on the left with, one on the right, reading a short paraigraph at the top 
of a page and answering questions about it in frames below, reading a sentence 
and selecting another to which it is logically related, or listening to the 
stimulus word being pronounced by tape recording and selecting that word from 
three choices on the test sheet. The variety of different formats is illustrated 
by the blank format sheets in Appendix B. - 

General and Specific Ob jectives . The reading tests were designed to measure 
performance on ten general objectives. Each general objective was further 
broken down into 3 to 21 specific objectives (listed in Appendix c). From 
these specific objectives were derived the 909 test items that constituted 
the 21 reading tests. Each reading test contained from 27 to UU items repre- 
senting from four to seven general objectives. Figure U shows the item break- 
down by test and general objective for the original tests. 

The selection of specific objectives on any single sheet was" governed by the 
vocabulary and story content in the immediately preceding lesson.. For instance, - 
fu sheet from test numbe^ 16, all stimulus words are taken from 

the three stories that pupils have most /^recently completed. Since each item 
requires the child to discriminate an initial consonant sound, all items are 
classified as part of a single general objective. However, each vocabulary 
word from the immediately preceding lesson begins with a different consonant, 
so that SIX different specific objectives, represented by those consonants, 
appear on the test. The complete brealtdown of items by specific objective 
for the original reading tests is shown in Appendix D. . 
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ol the. ^ current lesson, ar^d where items were thus wide.ly distributed over 
specific objectives. In such instances, the general objective was treated 
as a test, and a resulting prescription covered all of the specific objectives 

4 * A 0 4 *^ A ^ 



P r escriptions . Prescriptions were usually a single sheet of seatwork typical 
ot that currently used in classrooms throughout the city. These sheets con- 
tained practice items similar to those missed on the test, with tape-recorded 
instructions provided when necessary. Several sheets could be used in one day 
if pupils scores were low on more than one objective. 



Prescriptive seatwork for I^B was taken from several sources. Since teachers 
in the Los Angeles School System were already provided a set of seatwork 
materi^s keyed to the Ginn Basic Readers, that set was examined first. A 

included in that source for every story in the reading 
o^ies, utilizing the vocabulary introduced there. However, the skill most 

sheets was visual discrimination- -simple matching of 
. Development of phonic analysis or comprehension was scarcely repre- 

^hat pupils aLnost always 
’’^+0 • P®^cent or oetter in visual discrimination, so the school system 
materials were of little use . 



Many other commercial^ materials were examined, 
One was, again, poor representation of skills; 
that the pupils had not met in their readers, 
few appropriate sheets were taken from several 
keyed to li-B objectives. 



but they presented two problems, 
the other was use of a vocabulary 
With publishers' permission, a 
of these commercial sources and 






system's first-grade teachers for help in 

at sS needed. Two of the teaches worked 

Siw f the summer before itB operation, selecting and constructing remedial 
reoti’nn^^ materials that would provide practice on im objectives. Simple di- 
rections were typed on each sheet of work for easy teacher reference in «= 

one’foreLi°^blf tr ^^e first-grade classes in the fall, 

one for each objective represented on each sheet. 

Fmng^ys^. A coding system was devised for the follow-up materials. Each 

reoresertef “r the test n^ber folloved by the objective number of the 
uork represented. For instance, R2-0li03 on a sheet meant that it was for read inn 

nllmS^ SreLS'cl^srom ' TOe“?Sd«r 

exassroom. The folders were arranged in test and obiective 
cabinets in each participating classroom. Thus when 
t.^chers received printouts that called for rereLal sheet^by code nmSer 

t^i had only to remove enough sheets from the filing cabinet and use them’ 

With the group during their follow-up period. use them 




19 



10 May 1968 



14 



IM-3298/004/00 






O 

ERIC 



B. DATA PROCESSING 



Version I oX the i:-E information processing system has been operating since 
the first week in September I967. This version was used during the entire 
first semester with only minor changes. These changes and those made for 
the second semester are described in Section VI. 

Figure 5 shows tho '.lajor components in the initial data-processing system. 

The central component is the LUCID general purpose data management system, 
wliich operates under the time-sharing executive on the Q-32 computer. The 
LUCID system is used to store and retrieve the information that is necessary 
to produce repoirts of student progiress. Four other Ii^C program components, 

SSU, PROG, QUERY, and II-EOL, also operate ' under the Q-32 TSS executive. 

These programs serve as control programs for parts of LUCID; SSU formats data 
so that it is acceptable by LUCID, PROG produces regular student progress 
reports from data retrieved by LUCID, ,^d QUERY enables a user to directly 
address the LUCID data base to obtain information. IMSOL, an on-line program 
that composes reports, was added late in the semester. It is described in 
Section VI below. The fifth IMS component, labeled DIP in Figure 5, is a 
set of manual and mechanical procedures to convert data from original source 
documents (tests, rosters, etc.) to a binary tape acceptable to SSU. The 
next paragraphs give a more detailed description of the four initial components 

of It'S in the order in which they are used for processing student performance 
data. 

/ 

Data Input Procedures . These' procedures are divided into two sets. One set, 
which is necessary to prepare if.E so that it can be used by teachers, functions, 
only at the beginning of t^e semester. The other is required for the ongoing 
day-to-day use of the system. The first step in initial preparation of the • 
system for operation is to describe the structure of the course in terms, of 
objectived, tests, and prescriptions. This description is placed on punched 
cards and is read into the LUCID data management system. The second step in 
preparation is to store descriptive information, such as names of students, 
teachers, and schools; in the data base. These names are thus made available 
to the data management system for association with student response data. 

/ 

Collection of the descriptive information to prepare IMS for operation is 
accomplished through forms especially prepared for this purpose. These forms 
are filled out b^ teachers and by personnel who prepare the test . materials . They 
include a course description outline that lists general and specific objectives 
uy label; a test input form that associates individual items, teacher determined 
critericr levels, and prescription code numbers to objectives being measured; and 
a variet of /School, teacher, and pupil rosters (see Appendix F). The comple^ed 

transferred to punched cards and prestored on tape for entry 

/ 

/ 

The p- ocedures used for putting information into It-B on a day-to-day basis' start 
with Che/ test forms. One answer form for each test is filled out with the correct 



/' 



/ 
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FifPire j. T-IS First Semester Version. 
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answers marked. These forms, designated as belonging to student number 00 , 
are stored in the LUCID system and used as test keys for scoring student- 
produced tests. As groups of students take tests in the classrooms, the 
answer forms are brought to SDC by courier for processing with an IBM 1232 
Optical Scanner. Because of the novelty of this scanner use, it is described 
in detail in Section C below and in Appendix B. The data from 'the scanner 
are placed on electronic tape, and the tape, containing the day's yield of 
student test data, is then mounted on the Q-32. 

Scoring Summrizing, and Updating. The program to accomplish these functions 
IS written m the JOVIAL language and operates on the Q.32 TSS. The vITrtbI 
SSU reads the student response tape, interacts with the LUCm datl ma^^S^ 

test^item SSu" III ^eys (the responses for student 00),* and scores Sch 
test Item. SSU next computes the percentage of correct items for each maior 
'-o.,ecti/e included in the test and sends this information to LUCID. The ^ 

+ specific objective is also countL and 

4.U these tasks are accomplished for each test 

t ®rPfir+ is operated and the LUCID data base is changed ’ 

its tSks! performance data. At this point, SSU is finished witf. 



Generation of Progress Report. PROG is another program written in the mvTAT 
S-32.-^SSU has bean operate^d I InSrtlSa 
1 hp Aa+a ^ ^ group oi students, PROG is used to retrieve and format 

the data necessary to produce a progress report. PROG, encapsulates the audr- 
retr^-^p a e [unction ol LUCID, using this program to accomplish the required ^ 
retrieval tasks. This system design feature makes it a relatively simnle - 

“ is' optiinally! 



“onTfeasLr''”^^ "" f of tests 

e^th^t J '■“ cldtsroom before the next class meeting 

so that the teacher can use the information to pace her groups recroun hPi- ^ 

Sst^ uLoverld by t^^ 

Wfa semesteJ^ 6 Is an example of the progress report used during the fall 

The report is oriented to a specific 'group taking a soecific test Ou a 

'^£e~ ir? T"U%”^rre-'^5 raf-- 
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BUILDIMG WASHINGTON DATE II/06/67 

TEACHER HELEN ROBERTS' 

B1 READING BLUEBIRDS 
TEST 9905 GREEN BOOK, PP. 3-llt 

THIS GROUP DID WELL ON VISUAL DISCRIMINATION, SPECIFIC WORD CONFUSION, 
PilONICS, FINAL SOUNDS. THEY NEED MORE PRACTICE ON WORD RECOGNITION, SENTENCE 
COI-IPREHENSIONS. (ACTIVITIES R 5-0203, R5-090l|-. ) 



GROUP REPORT 


SCORE 




CU1«E- 




THIS 


ACTIVITIES 


LATIVE 


OBJECTIVE 


TEST 


RECOW-ENDED 


SCORE 


VISUAL DISCRBIINATION 


88 




93 


WORD RECOGNITION 


91 


R5-0203 


94 


SPECIFIC WORD CONFUSION 
PHONICS, INITIAL SOUNDS 


100 




■ 96 
95 


PHONICS, FINAL SOUNDS 
PHONICS, MEDIAL VOV/EIS 


97 




97 


STRUCTURAL ANALYSIS 
WORD COr-lPREHEI'ISION 






91 

98 


SENTENCE COMPREHENSION 
PARAGRAPH COMPREHENSION 


83 


R5-0904 


89 


GROUP AVERAGES 


89 




95 


INDIVIDUAL REPORT 




SCORE 


NUMBER 


cu^^u- 




THIS 


OF TESTS 


lATIVE 


NAI-E 


TEST 


TAKEN 


SCORE 


JEinff JONES 


100 


5 


100 


GREG ^ARSDEN 


100 


5 


98 


MARTHA COIE 


100 


5 


94 


BETTY PARSONS 


97 


5 


94 


DIANNE HOOKER 


91^ 


5 


97 


VIRGINIA DOUG IAS 


91 


3 


92 


SUSAN DRAKE 


88 


k 


95 


GERALD EFFINGHAJ.l 


41^ 


h 


78 


BRENDA ASH 




1 


100 


ICAREN ROGERS 




h 


98 


Ficure 6. Exajnple 


of Progress 


Report Produced by IMS. 
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those objectives included in the test. Note that some objectives are not 
covered by this particular test.. The second, column, *'ACTIVrPIES RECOMMENDED 
gives the location of prescriptive follow-up activities that the teacher can' 

-use to exercise specific objectives on which the group failed in this test 
Prescription is based on performance on specific— as contrasted ..to general- 
objectives. This is illustrated in Figure 6 ,, in that even though the Bluebirds 
^rforaed gererally well in "WORD RECOGNITION'’ (9I percent correct), they failed 
o a am criterion on one of the several specific objectives included under this 
category. The system sensed this failure and recommended a follow-up activity 
o exercise the specif ic -objective. The final columh, "CUMULATIVE SCORE," shows 

responses made by all Bluebirds for each general ob- 
tests taken by the group. Note again . that one obiective-- 
PHONICS, I 4 EDIAL VOWELS "--has not yet been tested. 

Part B of the report gives information on the individuals in the group. This 

tes-^^°thP pupil's name, his percentage of correct responses on this 

teso, the number of tests in the- sub ject that he has taken, and his cumulative 
score over those tests. This section can be used to assess the appropriateness 
diff'erantiT?''^ li"" ex^ple, "GERALD EFFINGHAM" is performing distinctly 

rifin candidate for reassignment to a "slower" 

for Detailed I nformation . The QUERY program is provided 

oijllP LfTr ! function of the LUCID system. This function is named 

QWP and can be requested from a console to interrogate a LUCID data base inter- 
actively. On the assumptions thai teachers (l) required defiled and specific 
perlormance data on individual students, (2) could learn to use a sllIToo^lnd 

•LiTOuSi ind information through a teletypewriter station hooked to a 
computer, and (3) would take their free time to make inquiries a teletvnpWr^■^P7- 
was instauad at each of the two schools where is "Smw. 
enalles a user to dial the sjc Q -32 over a regular tele^neuie ^Sat So^iSs 

nc S caiilortSe'l^'Tf 5 ). The user can request 
f *he INB data base, and enter requests for information The 

of SeT?r inro'-»«tlon, and pr™?r« ouf 

! telet^writer at the school. Using this program, a user can obtai^an 

le'r infinite variety of listings. Classroom rosters, group rosters individual 



c. 



AUl’OMATED COMPUTER INPUTS : THE IBM SCANI'JER 



student 

as new tests and’rp+pf+c continually being added to this file or revised 

f J retests are given throughout the semester. Teachers need th^ 
leedback on student performance almost immediately: therefore a rnp^h 

nu"LT‘S1h0rf 5 “' a’ 

quired, so that t„e computer could evaluate the studenta' responaes and identify 
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any necessary prescriptions . The Instructional Management System uses^t^lie 
IBM 1232 Optical Hark Page Reader in a' somewhat novel way to facilitate this 
transfer operation. // 



Some ot)tical- scanners are designed'to interpret hand-written characters, others 
to interpret characters typed or printed in a special font, while others simply 
interpret horizontal or vertical pencil matks (bar codes) marked on an input 
document. The bar code type' of scanner was selected for II® use because this 
type has a. relatively low cost, is very reliable, eind requires the least skill 
in marking the response forms. The IBM 1232 scaimer has these characteristics 
plus simple operation ^d fairly rapid delivery time. Input documents for the 
1232 are in staridart hg 11 inch size and cem be marked with a common 
pencil. By operating off-line (i.e., not directly linked to a computer or data 
processing system), the 1232 remains independent of computer schedules and large 
system breakdowns. The 1232 can read up to 1,000 mark positions on each input 
document (see Figure 7). The positions are arranged in rows of 20, with spaces 
between each set of five. Each row of 20 is logically divided by the machine 
into two scanner words of 10 positions, and each scanner word is divided into 
tw6 scanrier segments of 5 positions each* 

operation of the 1232 are relatively simple. Documents are 
led into the 1232 face up, one at a time. The document passes under a read head 
(see Figure 6), where light from 20 light sources is bounced off the sheet and 
lack into 20 photo-electric cells. When a mark is made in a mark positibn, the 
pencil lead absorbs a certain amount of the light, and the output of the corre- ' 

./ spending photo-electric cell is reduced. It is the reduction of electrical output 
01 the cell that is sensed as a mark. 



2,000 documents per hour, .but because it is attached to 
an IBM 5.34 kejTJunch, it is limited by the keypunch to 600 fully punched cards 
per hour. As with most bar code scanners, £he input documents must be very 
precisely printed. Most of the tolerances are plus and minus five-thousandths 

^ Fainted instructions, mark position 

stimulus materials must be printed with highly reflective inks 
in order not to be read by the scanner. 

needs, the scanner input documents had to be modified to meet certain 

weJS Lre'^effe"t"*^ tests. Prior trials had shown that some formats 

r ll presenting certain kinds of test items than ^^re others. 

It was essential that the format or Ipcation of mark positions on the document 

to°be t o standard IBM mark positions appeLed 

io ll \ mark positions Seded 

docuLnt ^ ' enlarged, and decreased in number to make a simpler response 



A solution was reached by arranging several dii’i-erent 
basic 30-row IBM scanner sheet. Each format had fewer 
centered on the positions of the basic IBM sheet. The 



response formats on a 
, larger mark positions 
identification of each 



O 
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figure 7 * Mwk Positions on Standard Scanner Form,- 
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Figure 8, Operation of Optical Scanner. 
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"legal" Il-lS mark position on a given sheet was indicated by a preprinted, 
machine readable, format code placed in the last row of the input doca-aent. 
Although the scanner was programmed to read all 30 rows, 60 words, 600 mark 
positions on the 83 by 11 sheet, only 9 to 35 of the mark positions of any 
given sheet were printed on the sheet itself or interpreted by the computer 

program# £ach of these ma rk positions described an oval ax*ea around 
the standard 13-1 mark position and approximately four times larger. The larger 
mark positions were broadly spaced across the sheet, so if a nark extended 
beyond the "legal" mark boundaries, there was little chance that a punch would 
he genezated in the next position examined by the processing progrEim# The 
scanner output deck was read by the computer in a binary mode, so that a multi- 
punch geneii'ated by a long mark could be examined for the presence or absence of 
the punch in the "legal" IMS mark position. Designation of "legal" mark posi- 
tions allowed great freedom in sheet design. Items could be arranged horizontally 
or vertically, and/or across several scanner words. 

To organize the materials so that first graders would be sure which stimuli went 
with an item, a pink stripe was printed over every other item presentation area. 
The resulting pinlc and white stripes on the test page served as a handy reference 
and check in the sequence of instructions to items in the test. Instructions 
like, Now do the next pink row...," assisted the student in keeping his place 
in the test. 



The scanner was programmed to punch out one card for every page of the BIS tests. 

To associate each of the pages of a test with a given student, a master mark 
sheet was used as a cover sheet. Students wrote their names on the cover of the 
test booklet, and a unique student number was coded into the four scanner words 
on the master mark sheet. The cover sheet did not generate a separate punched 
card as did the other pages of the test, but stored the coded student number in 
the scanner memory and punched it into four identification columns on the cards 
generated for each subsequent page. Each new cover sheet encountered in processing 
replaced the previous student number in the scanner memory and punched its code 
into the subsec^uent cards* 



Maintenance of the test page order was essential to the processing, items on 
'the test were not identified to the scanner except by the order in which they 
were read, so if page five was accidentally processed before page four, the 
Items on both pages would be mis-scored. To insure that the sequence of the 
items would remain fixed, the pages of the BIS tests were stapled to-ether at 
ne left-hand comer; the staple could be rapidly removed for processing, 

r the pages could easily be turned when the students were taking the test. 

In addition, the pages were clearly numbered in large numerals in the Ic-.'er 
right-hand corner. 



Details of procedures used and problems encountered in production of the machine 
readable tests for first graders are given in Appendix B. 
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D. CIASSROa-; PRESEriTATIOU SYSTH-I 

The teacher's classroom schedule in the demnstration schools, though flexible, 
is built around 20 -minute periods for vrorh with each reading group. The 
test exercises are designed to be administered during a 20-minutc follow-up 
period irtiile the teacher works in the circle with another leading group. Each 
child in the follow-up group works on the same test with directions presented 
by tape recorder through headsets. In the original design of presentation 
equipment, the directions were read for a single item or a page of similar 
items. Then the tape recorder stopped until each child pushed a button to 
signal that he was ready to go on. Thus, the tests were paced by the slowest 
child. 

An individual presentation system that could provide this capability was the 
audio-frame system, produced by Appleton-Century-Croft, using a Norelco cartridge 
type tape recorder. This system stopped automatically after three seconds of 
silence and was restarted by pushing a button. The system was expanded into a 
group system by adding a set of relays corresponding to a group of students— 
the last student to respond completed the circuit and restarted the tape. 

The relay control box contained a set of switches, corresponding to the student 
positions, that allowed unoccupied positions to be selected out of the control 
circuit. Another switch allowed the teacher to play a separate sound source 
through the system — unaffected by the relays, the tape recorder, or the t>ower 
supply. 

The students listened to the instructions through headsets plugged into boxes 
in the students' carrels. Each box also contained a mini bulb (28 volts) and 
a clear plastic response button. When the child pushed his "ready button," the 
light went out; when the machine restarted (following the last student's re- 
sponse), the light came on again and remained lit until the student again nushed 
his ready button. 

Recordi ng System . Because the playback system. was designed to stop after three 
seconds of recorded silence, there was a danger that it might stop unintention- 
ally at times vhen there was a pause in the voice' recording but when no written 
pupil res^nse was desired. An example would be after the instruction "Turn to 
page two." To avoid this problem, the first tapes were recorded directly on a 
tape-cartridge recorder that had been modified by the addition of a switched 
oO-cycle tone generator. This tone kept the recorder running. When a stop was 
desired, the tone generator was switched off. After the master tanes were 
completed, four copies of each were dubbed. 

Tnis initial mode of operation proved unsatisfactory: (l) The 60 -cycle tone 

intruded on the voice and seriously degraded the voice quality of the master 
ta^; (2) the cartridge recorder was of sufficiently poor quality that the dubs 
made from it (the distortion of the master is roughly squared in the dub) were 
of TCry poor quality; (3) as a result of mechanical, human, or distortional 
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errors, spurious steps occasionally appeared on the dubs; and ( 4 ) the svatch 

controlling the bO-cycle tone generator anno-or.ced itself vith an annoyin' se-ie- 
of pops and clicks. ^ 

Most of these probleris were overcone by recording master tapes on a high fidelity 
roel-to-i-cel nacniiie and by development of a '/o ice -actuated, switched-tone 

eng^eering. With this latter device, the tone was injected 
automatically when the ’.^ice stopped, but a silent input switch could be onerated 
manually to turn it; off when a stop was required. The resulting dubs were* of 
much Higher- sound quality than under the former method of recording and were 
of its anno^^'ln^ pops. 

S^udy Carrel Desi(=p. The initial design for carrels provided for r^sonite 
panels attached to aluninam cnannel, providing spaces approximately I8" wide b- 
deep. T!ie ready boxes were mounted on the center nakition of the carrels'" 
-eav.ng room on the desk to place the test booklets. The carrels were to be " 
P^aceu on tables provided by the ^s Angfeles City Schools, and two sizes had 
ceen planned: student tables, 19I" x 40 ^”, and library tables, 3 ’ x o'. The 

substituted; this unfortuustely 

reauced the size of the study spaces. 

^cause tnere were only two classroom groups for mathematics (coiSared to three 

of -IthfiTf]' positions might be required for* presentation 

iLT i ® ^ arranged by adding an extra nair of Jacks to a 

?? ? tables. An "extension^’ was connected 

+ allowing the use of at least four more headsets for students wtio 
could listen at adjacent tables, but who had no "ready buttons." 



U 



EVAUJATION OF FIRST SEJ»n^-qTER TRIAL 



Tnere w re two major considerations in evaluating the fall semester's E-IS 
aemonstration. Since the primary objective of this year's project v-as to estab- 
lisu an operating system, the first consideration was that of practicality- -the 

»ov functlcued. Did the teaS“Si^l=ter 
tne tests* Did classroom tape recording systems operate adequately* Was the 
dcanner versatile enoueh to handle the first grade's' narks? Dirconutl^lr 

generated aeeuratky and or. 
tine. Did tne interactive query capability operate? The available data which 

,trth\‘^rf?i^s..:?^ 

Thonq-h the pur^se of this year's developmental effort was to build an one-atln~ 

‘^<=‘ithing clearly was only the first nmbler.. Se ‘^-rte-'s 
helne; ?de^fv oSe n reflected BIS's own perfo^nr^ " 

aspect ^^ar ^a tS SaL^^^ 
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situation US6 H-.S to facilitate pupil achieveirsnt of course objectives? Data 
collected to answer this question, both negati'.’e and positi\-e, should guide 
de'/elopr£nt of nore refined procedures to inpro\-e classroon remediation of 
instructional difficulties. These data are also discussed below. 



A. .’ECILArnCAL OPERilTION OF THE 3YSTE11 



Se\’eral sources were consulted to obtain data conceiming the oneration of I1-!S. 
One source consisted of descripti\'e notes kept by project personnel as they 
observed BIS in operation. Records were kept by five project staff nenbers: 
one who served as teletype operator and interacted with teachers, a second who 
had responsibility for classroom recorders and headsets used in nresentation, 
a third who conducted scanner operations, and two programmers who operated the 
computers to produce output data. A second source of information consisted of 
classroom observations made by the school liaison person. She observed each 
classroom once a week during the reading period, following different grouns 
from week to week as they progressed through the entire reading time cycle — 
one-third with the teacher in the reading circle, one-third doing follow-up 
seatwork, and one-third doing independent work. A third source of information 
was the regular classroom performance printout. The final source consisted of 
teacher observations and attitudes obtained through a cj^uestionnaire# 



Test Administration. The first operational question concerns administration of 
tne tests. The reading tests were designed to be administered after e%rery third 
story^in the reading books. In general, the teachers kept close to this schedule. 
\n\en they were absent from the classroon due to illness, they fell behind with 
tne exercises; however, they attempted to catch up quckly when they returned 
Iwnile still giving the recommended prescriptions). There was considerable 
variation in the number of tests completed. The most advanced of three reading 
groups in one class completed all 2? of the reading tests, while the top group 
in another completed 24 tests. At the other extreme, two slow grouns comnleted 
only the eight readiness tests and four reading tests. The eight remaining groups 
in the four classes progressed to points varying from test 7 through test 167 
Positions of the groups in math ranged from test 2 to test 13 of the l6 rath tests. 



Thus, the tests were generally administered carefully and in sequence b-"- the 
teachers. There were no major administration problems, but some confusion 
existed at times. One problem in test scheduling was keening track of the 
various groups progress through the tests. The difficulty resulted from differ- 
ences in the different groups rate of progress and from the fact that a teacher 
sometimes received two duplicate printouts, one a day or two later than the other 
Duplicate printouts occurred when a child took a makeup test, or took^ tTst liZ' 
the a^ed ^a°^ causing the computer programs to generate a second report with 



Group nixups 
the children 
the multiple 



were a frequent occurrence in the beginning of the semester when 
were Just learning the concept of grouping. Partly as a result of 
reports and the confusion in grouping, one teacher skinned six group 
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tests and auplicated another three. Two other teachers skinned a total of 
four group tests and duplicated one. This, however, is a s^ll n'orbe^ i- 
relation to the total number of tests administered. 

Another relatively r.inor problem in test administration concerned tl-nv o^' 
tests. From earlier tryouts of the tests, their timing vtis tailoi-ed to f ■: t 
Lnto a 2j-minute period. However, during the first eight weeks of nresentation, 
a stop-tape sptem was used that presented instructions, then allowed each chi^d 
to signal witn a button push when he was ready to go on. Thus, directions -■'or 
new responses were not presented until the slowest child was ready. This s*ste- 
..aae timing of the tests -/ary more than had been anticipated. To' illustrate a 

recorded by the observer, varied in length from 23 to ' 
j minutes. Otners \aried from lU to 21 minutes and 18 to 26 minutes. The 

w?th^el?r flexible, is built around 20-minute neriods 

effectively plan lesson time. Occasionally an exercise was interrupte 
to ta/.e tne cnildren out to recess. i-cx-ruptt. 

^ second part of the operations to be considered 
r^fn classroom presentation system. Obser-zation of classes 

introduction of a management system led project members to belie%-e 
•■o- children could not work independently without having directions 

* * single iteiM or groups of similar items read to them Lmmediately before they 
. to respond. The tape recording system was designed to SSl'ue ^ 
toots self-administering by providing the needed directions. Teachers occunied 

then be Intermpted by confCS 

the goal, an effort was 

child^L^to^the'^foir^ independence the recording system afforded to 

n -ber o- ttoes classroom observer carefully recorded the 

reaLng Jire^e! ^roup interrupted the teacher -working in the 

Child-caused malfunctions, mechanical difficulties occurred- 

ri £riHr^^ -ki: 
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These proble.-ns '.rere much nore apparent at the beginning of the semester; the 
rear, number of obser-.-ed interruptions at that time ranged from 2.S to U.l oer 
20-r.i.nute reading period in the '/arious classroor.s. Later in the semester* 
with the introduction of more simply designed equip.tent, this figure” dropped 
to O.S and 1»0 interruptions in the classes obser’/ed. The sane grouos of 
children, when engaged in other Independent activities such as painting or 
wording puzzles, interrupted the teacher nore often than they did when working 
in the B-IS listening center. 

The teacher also spent less tine gi\'ing initial directions for BiS test sessions 
tha.n s.ne did for otner activities. The liaison person observed 38 BIS test 
administrations and 32 equal time periods when the pupils were engaged in other 
activities. On 10 of the 3S test ad,-ninlstratlons and on 23 of the 32 comparison 
occasions, the teacher spent two minutes or more providing directions. Thus the 
presentation syste.m facilitated classroom management by providing self-administered 
work for pupil follow-up. 



Optical Scanner, .q third consideration in e-/aluation of system ooerations is 
that of the optical scanner as an input device for the programs, 'were first 
graders capable of learning how to respond on printed forms in compliance with 
recorded instructions? Could the optical scanner process documents that were 
bent, folded, and mangled in testing, marked with oversized primary’- pencils and 
comer-cut in preprocessing? Early testing and one semester's practice indicate 
that the answer to these questions is yes. From the earliest reading tests, the 
first-grade students from both schools demonstrated that they were caoable of 
.Taking good, machine readable marks. Later in the semester, many students 
noticed the boxes on the cover sheet for student identification namber, created 
a four-digit number for themselves, and -wrote it into the space provided. 
Although these numbers were not used in the test processing, mhey did indicate 
that the children could probably be trained to mark a designated identification 
number on the sheets. This would reduce the preprocessing time significantly 
and would reduce confusions that occur when two students with the same first 
name do not write their last names on the test sheet. 



But there were some problems in scaamer use. For instance, as the semester 
progressed, the quality of pupils' rarks seamed to deteriorate rather than 
l^pro^. Evnen students in the top groups would slip into making careless marks, 
owar t|^ end of the semester, it was necessary for the teachers to re-emnhasize 

good marks and to review the procedures for making them. 
With that erapnasis, the marks did improve. The most serious problem in the actual 
processing of the sheets was jamming of the sheets inside the machine. The 
incidence of this problem in most scanner operations is normally low, but when 
tne documents are bent and cut on the left corners, folded and cnrroled, or' when 

smears from sticky fingers works to 
the sneets of a test booklet together, sheet jams become more frequent. 

To recover irom a sneet jam, the operator has two recourses: he can try to 

force the documents through; if that fails, he must pull the sheets from the 
feeding mecnanism. Often this last recovery' procedure tears the input document. 
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12 the input doou-oent is tom, a duplicate must be r^de. For this e'.-entualitv 
a cortplete f*xle of* blaihc fonts was Kept by tne scanner* *vinen a sheet 

jaotted and vas tom, another sheet with the sane fomat vas taken fron tne file x 
and the responses of the original sheet copied onto the duplicate. To ore.^nt 
further Jans, whenever a sheet was torn in tne itachine the docu-tent feeding 
patn vreis checked for snail scraps of paper. 

Another interruption in the processing occurred when a sheet would "select." 
i-ecause the sequence of the sheets is essential to the interpretation of tne 
data the computer, a sneet that fails one of tne machine checks is 'selected* 

out of the processing stream end stops the processing of the data. It is 
partially for this reason that the ttning mark areas and the format codes are 
checKed out not only by the machine, but also in the preprocessing procedures. 

It WHS essential that these areas be read and interpreted correctly, but it was 
also important that they pass these tests in order to a'/oid being selected out. 
’^ner. a sheet uhs selected out, processir^^ had to be stopped, the sheet corrected, 
the incorrect data card removed from tne deck, and the sheet renrocessed. 

■ecause of careful preprocessing, sheets were rarely selected because of’er-ors 
in these areas. 

T:ie 1232 operation and related checking procedures did demonstrate several clear 
advantages. The whole scanner operation could be accomplished in one-third to 
one-fourth the time required for manual keypunching of test data. Four to seven 
tests on from 2UO to A20 individual sheets of items could be D'»'e- 
processed and scanned in If. to 2 hours, with two people working on tne operation. 
«t one point in the semester, when a brea’Kdown of computer programs necessitated 
co-ng the 'work by hand, it took the same people twice as long .merely to score 
c.ne tests without providing any information on how the nupils nerformed on 
different objectives. 



The number of tests scored for the demonstration does not begin to tax the 
capacity of the scanner. Coupled with the IB.M 53^ keypunch, the 1232 can'nroce 
to sneets an hour, and thus could acco.mmodate a much larger syste.m.* 



ss 



^omnuter Processing . The next step in the operational sequence is the 
01 co.mputer programs to produce printouts. The design specifications of th^^ 
system required that a report be generated for teachers the morning after each 
test was administered. It was to contain a summary of .vrouD pe^'^o^ance ' y 
general objective, a listing of reconr.ended remedial wo?k wrien the'grouD score; 
lelow' criterion, and an individual listing of pupils’ test scores. 



Oftc 



blank 



vnere were several minor problems when E.'.3 programs were first instituted, 
these were due to lack of sophistication on some staff me.mbers' nart about 
computer progra.mming. For instance, the test writers left some item snac-s 
on a page and, quite logically, numbered only those spaces that -.taine, 
stimuli. However, the computer counts evei^r space, so items had to be .manually 
renumbered. Or, occasionally, a teacher would get a new punil, give him a test 

analysis before she had entered the pupil’s nam.e and number 
-n tho data base, nlso, stai f -etibers doing tine preprocessing assa-.ed that w.nen 
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a' child aarked all three responses to an iten, it would simply be counted as 
incorrect. Howe-.’er, three marks generated a specific type of card punch, 
leading to an error in the QUU? program that terminated computer processing. 

This problem was remedied in the computer programs themselves. 

The problems described above were transitory, and the time and quality speci- 
fications for printouts were generally net. During the semester the four 
teachers werc^ provided with approximately 3OQ routine performance summary 
printou^"^for their reading and math groups. From September until December 
there ,were only six days when reports were late. 

/ 

Query System . The last operational component of the system to be discussed in 
e'/aluation is the interactive query system. A teletypewriter terminal connected 
to tne Q-32 computer at SDC was installed at each of the participating schools 
for on-line interacti-/e query on the data base. It was anticipated that the 
teachers would ha-;e questions that we^re not answered in the daily printout. 
Through the use of the teletype , questions such as how well a particular child 
performed on specific objectives could be answered almost instantaneously, 
because the teachers were not acquainted with the language of the programs, an 
operator w^as available in each school between 12:00 noon and 1:00 p.m. to 
interpret their questions and get answers from the data base. 

Early in the semester, the teachers were receiving so much more detailed 
information on each child than they normally received that they t’elt no need 
for additional data. Also, they were not fully aware of the information that 
was potentially available. To stimulate questions, the staff members generated 
sample displays illustrating the tjrpes of information the teachers could obtain. 
With this priming the teachers began to ask for specific information on 
individual children. 

Table 1 shows the number of times the query system was used and the nature of 
the inf'ormation provided during the first semester. ' . The most extensive request 
was by teachers in one school for scores for each student at report card time. 
Aside from this use for grading purposes, the teletype query was most requested 
for diagnostic information and parent conferences. 

&. TEACHER USE OF D-!S DATA 

Though II'S has not been tried, e'/aluated, and revised enough times to bring its 
mechanical operation to an optimal state, it clearly meets specifications for 
the first demonstration. Determining its effect upon the classroom is a more 
difficult problem. Ultimately, e'/aluation of any classroom innovation must be 
related to changes in pupil performance. However, E-IS only provides data on 
pupil performance and available resources; the teacher arranges conditions that 
facilitate pupil achievement of course objectives. Both components of the 
instructional system.., BS and teacher, must function adequately before a direct 
effect upon pupil performance can be expected. Until BIS was in operation, 
teacher use of its resources could not be in^/estigated. Now, after a semester's 
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Table 1. Use of Telet^'pe 





No 


. of TTY Printouts 


Provided 




Information Provided 
Via Teletype 


Teacher 

1 


Teacher 

2 


Teacher 

_ 3 


Teacher 

4 


Total 


Verify up-to-date rosters 


6 


5 


5 


5 


21 


Provide lists of objectives 


1 


1 


1 


1 


4 


Individual scores by major and 
specific objectives for 
teacher information eind 
diagnosis 


1 


2 


U 


9 


Id 


Parent conferences 






3 


2 


5 


Tests missed by specific children 


c" ^ 


JL 






2 


Dates for last group test 


1 


1 






2 


Test and major objective scores 
by tests for grading 




1 






2 


Totaj. 


11 


11 


13 


17 


53 
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trial, available data can be examined to determine how teachers used 
information and to generate hypotheses concerning new data, resources, and 
training that might facilitate teacher performance. 

Evaluation of Data Internal to the System . IMS not only provides management 
data for the teacher, but records many of her instructional decisions in its 
data base and carefully tracks student performance before and after the decision 
is made. Even with incomplete data collected this first semester, when all 
components of the system were under development, a post hoc analysis of system 
data clearly indicates the classrooms in which pupils consistently improved in 
achie\ement of objecti-'/es and those in vdiich they did not. To illustrate this, 
scores for three of the ten general reading objectives were recorded for every 
test that each group took. The three objectives with the lowest initial scores 
(phonic analysis, sentence comprehension, and paragraph comprehension) were 
selected because those objectives on which pupils began at near mastery level 
could show no gain scores. Table 2 presents the first and last in the series 
of scores for each of the 2o cases where a group finished at least three tests 
in one of the objectives. 

The data are most complete for teachers three and four. Eight times out of 
nine, the groups in teacher three's class either improved appreciably or stayed 
within the ceiling score they began on. Teacher four was less successful. In 
only two of nine cases did her pupils rise appreciably in score. In four cases 
where they were potentially able to rise, they stayed the same, and in three 
cases they fell measurably. The data for teachers one and two will not be 
complete until the end of the second semester, but they appear to be only 
partially successful. Achievement in teacher one's class rose three times and 
fell or failed to rise twice. In teacher two's class, achievement rose twice 
and fell three times. Teachers three and four are at the same school and hare 
apparently similar classes. Teachers one and two are at another school where 
they share adjoining rooms and also appear to have similar classes. 

The scores in Table 2 do not show the great improvement that might hare been 
expected in performance, but they must be exEimined in the light of other data 
showing the extent to which the teachers used tne IMS report information in ■ 
making their instructional decisions. These latter data suggest that the 
teachers as a whole did not use the H-IS data as much as they could hare. The 
teacher who did make greatest use of the data, teacher three, is the teacher 
whose students showed the greatest gains. The following paragraphs discuss in 
more detail the types of instructional decisions in'question, and the data 
concerning teacher use of the BIS report information. 

Teachers in B-LS classrooms divide their 30 pupils into three Rn^allp-r groups to 
present lessons. This allows Increased opportunities for individual attention • 
and for tailoring instructional sequences to children with more homogeneous 
needs. The teacher is limited in the kinds of day-to-day actions she can take 
within this instructional system. She can alter lesson pacing, moving more 
slowly when pupils are having difficulty. She can provide supplemental 
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Table 2. Initial and Final Scores by 
Teacher, Group, and Objective. 



Teacher 


Group 


Phonic 

Analysis 


Sentence 
Gomprehens ion 


Para»:rap.'. 

Conprehensio:-. 




iliga 


06-77 


67-69 




or.T 


Middle 


Uo-57 


— 







Lou 


UO -22 


— 


— 




Hica 


66-61 


"^ 7-73 ' 




T’.,'0 


Middle 


ol-Lb 


— 


- - 




Low 


21-42 


— 


--- 




. HiGh 


9 b -92 


93 - ■ 




TIIR2E 


Middle 


('. 1-97 ' 








I^w 


^ 0 — 


3 3 - i? 







Hic’a 


69-,)i; 


FOUR 


Middle 


/ 2-71 




Low 


6 L -60 
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instruction in the form of remedial sequences for the group. She can regroup 
children so that pupils with similar achievement levels are working together. 
Finally, she can provide individuai ihstrucdtiqn to bring a low achiever up to' 
his group's performance. Available data give Vome indication of how closely . 
BIS teachers followed information provided on progress reports in making these 
decisions . 

Figure 9 provides evidence that all of the teachers alter lesson pacing in 
relation to the achievement level of their pupils; the higher scoring pupils move 
faster and the lower scoring pupils move slower. However, the pacing seems to 
be more closely related to the school than to the actual performance level of 
the pupils. Classes of teachers one and two completed about the same number of 
tests, and classes of teachers three and four completed about the same number 
of tests. This would not have been predicted from the scores, since scores of 
pupils in teacher four's class were more like those in teacher one and teacher 
two's classes than those in teacher three's class. The mean number of days 
between tests for each class reflects this same school difference — 4.9 days for 
teacher one, 4.3 days for teacher two, 2.6 days for teacher three, and 3.4 days 
for teacher four. 

Although the teachers generally paced groups by their test scores, the relation- 
ship of days between tests to specific le'/els of achievement was not always 
consistent. Table 3 shows the mean number of days between tests by score level 
for each class. It peems logical that when a group makes a high score on a 
test, less time will be spent in remedial work and preparation before the next 
test than when the group makes a low score. If the teacher followed this logic, 
as the scores in Table 3 decrease , mean numbers of days should increase , with 
no reversals in' the relationship. This was the case only for teachers two and 
three. 

The second type of decision on which information was collected is that of 
providing supplemental instruction for groups that are not achieving particular 
objectives, IMS provides for such instruction through the prescription feature 
of the printout, \4hen a group does not meet the criterion score on an objective, 
specific follow-up seatwor’: is recommended to give additional practice in that 
prescriptions are to be done by the groups independently, following 
worK in the reading circle with the teacher. In addition to IMS prescriptions, 
the teacher plans exercises of her own to strengthen reading skills and routinely 
gives them to the children. 

In general the printout prescriptions were followed promptly. The teachers 
stated that when they-received the printout they pulled those prescriptions 
listed from the fileo, set them aside, and used them during the appronriate 
period until they were gone. \Vhen large numbers were listed (for slow groups, 
for instance ) the children could not always complete them in one day, so there 
a carryover. Table 4 shows the total number of prescriptions recommended 
for each class and the number that each teacher administered. 
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Figure 9- Nu-nber of Tests Completed and Cumulative 

Scores by Reading Groups 
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Table 3 * 


Mean 


Number of Days Between Tests by Score 
Level for Each Class. 




Test Score 


One 


Teacher 

Two 


Three 


Four 


90-100 


k.l 


3.0 


2.2 


2.5 


80-69 


3.1 


3.9 


3.5 


3.0 


70 -Y 9 




1 


3-5 


U.2 


60-69 


3.2 


5.0 / 

/ 


U.O 


2.3 


50-59 • 


8.0 , 


5.0, 


No entries 


5.0 
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Table U. Teacher Use of Prescriptions 



Teacher 


Total Number of Group Prescriptions 
Recommended Not Gi-ven Percent Gl-ven 


One 


100 


19 


81 


Two 


93 


5 


95 


Three 


■ 84 


9 


90 


Four 


108 


29 


73 


TOTAL 


385 


62 


84 



The total number of prescriptions recommended per class ranged from 8U to I 08 . 

There were 200 different sets of follow-up materials available in the files, 
each in sufficient ciuantities for the three greups« Of those materials pre- 
scribed, the number not given ranged from 5 to 29 . The percent given ranged 
from 73 percent for teacher four to 95 percent for teacher two. When questioned, 
the teachers said they \rere giving everything prescribed, but there was a small 
lag at the beginning \rtien the system was getting underway , and there appeared 
to be oversights along the way. There was also additional use of the nonpre scribed 
follow-up materials keyed to objectives. Teacher three used them extensirely 
and systematically— U 9 times in addition to the 84 that were prescribed. Teacher 
one did this also, but only 16 times beyond printout recommendations. 



Another measurement related to supplemental instruction was obtained through 
observation. It wb(s a' tabalatlon "or minutes' spent- jjrovidihg'dire'ctio^^^ for any 
period of independent, academically oriented work. The teachers differed some- 
what here. On 42 percent Of the observed occasions, teacher one spent two 
minutes or more giving directions before pupils started independent work; teacher 
two spent that much time on 39 percent of the observed occasions, teacher three 
on ol percent of the occasions, and teacher four on 4? percent of the occasions. 



The third kind of instructional decision on -vrtiich we ha-ve data is that of 
regrouping children. Since every W>S printout lists the rank of each child in ' 
the -teacher s group, it would seem logical for top-ranked children to be moved 
to higher groups and bottom-ranked children to be moved to lower groups.' The 
change ^ta show that this was not usually the case, except for teacher three 
who quite consistently followed ranking information in changing children to 
ifferent groups. Forty- four changes were made during the semester by all teachers 
Out of nine changes made by teacher one, in only one case was the bottom-ranked 





i 



10 May 1968 



37 



I 

H.l-3298/004/00 



child moved down, and in only two cases was the top-ranked child moved up. 

For teacher two, in only one case out of ten changes was the top-ranked child 
moved up, and in no case was the bottom-ranked child moved down. Eleven of 
the 14 changes made by teacher three were to move top-ranked children up or 
bottom- ranked children down. In no case out of seven changes did teacher four 
make this kind of move. Four children not included in this analysis were 
removed from classroom group^for special treatment. 

The last classroom decision activity to be considered is that of providing 
individual attention to children whose scores are low. Regular IMS printouts 
identify low achievers by listing, total test scores and ranking children within 
the group. For a teacher to learn on which objective a low-scoring child 
needed help, she had to query the data base via teletype Only 16 times was 
this done by all of the four teachers^- In no case did the teachers follow up 
these requests by asking what prescriptions were available to remedy the low 
achievement. 

The observer also tried to assess the individual attention each teacher 
provided by counting the number of times each teacher interacted in any way 
with a low-scoring pupil during reading circle instruction. The range for 
20-minute period was from 1 to I 5 single interactions, but the only clear 
pattern was one related to school. Teachers one and two, whose pupils tended 
to move more slowly through the sequence of lessons, interacted with low- 
scoring children more often than did teachers three and four. This lack of 
emphasis on the individual child is not entirely unexpected. Classroom pre- 
sentations ■ are always in the group mode, so it is logical that the teacher's 
enphasis more often concerns the action to take with the group rather than 
with specific children. To effect a change in this attitude, she will probably 
have to receive automatically more individual information and concrete 
suggestions for its use. 

In addition to analyzing performance scores on tests that are internal to IMS 
and investigating teacher use of system data, two other techniques were used 
in evaluation of UvlS. One involved administration of an external criterion 
test at the end of preprimer instruction to IMS pupils and to pupils of 
comparison schools located in the same general area. The second was a ques- 
tionnaire administered to teachers twice ■ during the semester in an attenpt to 
gain information on attitudes as well as system operation. 

Comparison Test Responses . There were two parts to the comparison test. On 
the first day children were tested with items that included only vocabulary 
they had been es^osed to in their Ginn series readers. There were four kinds 
of items: visual discrimination, word recognition, oral word comprehension, 

and sentence comprehension. Visual discrimination required simple matching 
of printed words; word recognition required the pupil to select the printed 
word to match a word that was read to him orally; in oral word conprehension, 
the pupil selected a picture to show he understood the meaning of a pronounced 
word; and in sentence conprehension, he read a sentence and selected a nicture 
that illustrated its meaning. 
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Since the premise of any reading instructional program is that it will train 
the child to read all words rather than just those he has met in class a 
second day's test was administered. Its purpose -was-to determine how well 
pupils in IMS reading programs could decode new words wtien the words involved 
phonemes the cnildren had been trained on. The test included the s.ame four 
kinds of items as the first day's test, but the child had not encountered the 
words in the classroom. Instead, the initial consonants and final porliions 
of words from the first day's test were separated, tinen recombined to form new 
words for the transfer vocabulary. For example, the words "fast" and "look" 
appeared on the first test and the transfer wcabulary wrd that was tested 

VQS X&SX* 



groups completed the preprimer stage of instruction during 

groups were tested was especially 

critical. Therefore, no statistical analysis was performed on the data, but 
the scores are presented in Table 5. IMS groups include pupils 'in the re^ar 
program of BIS classrooms. Comparison groups include punils in 
the Ginn program without the management system. The Ginn/i^ius supplement croun 
was drara from a class in which, by teacher report, Ginn" materia if were supple- 
mented by regular phonics drill— direct training on the transfer test taskf 

O'"* items involving Ginn vocabulary; a 

statistical test would surely show no significant differences. Perfon.iSce on 
was mch lowever, hov/ever, and some differences were apparent. 
BIS pupils performed better than Ginn-only comparison groups on 
differences was from four to nine percentage 
+ 1 1 Tj^ ^ task for which neither group had received more than indirect 

training. Horever, the Ginn plus supplement group, which received direct 
raining on the task, was obviously superior, being from 2 to 31 points higher 

trei^iifon°f r results suggest a hypothesis: combining direcf 

training on a tas.i and a management system to assess that training's effects at 
each step of the way would provide most effective instruction. ® 

Questionnaire Responses . The questionnaire administered twice during the 

about how teachers reacted to the ^management 

1 vorable--difficult to interpret except in a descriptive sense. 

The teachers unanimously agreed at mid-semester that the children seemed to 

enjoy wrlcing with the Instmctlonal Management System mom t^ noi^l 

Slven for this that th^ ““ho^es anHiher 

irtorplur" "men”th^‘'‘’'''°"* “T?” ® ntollenge 

xne pupllo. When the same question was asked at the conclusion of the 

t!:“sa!rtr'^ r 

^ ^ enjoyed the new procedures about the same as con- 

in one"cla^''icaSf toref Wrd'^L''e^^^ ^ f children 

tested every day. There h^eln^'e^l^nf fr^rr^hl^rrUe^irS 

fft-f SavrtSrjnrdJ'f'JieTu ^ 
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The teachers' reactions to IMS were favorable as indicated by some of the 
following statements. "It has helped me in my work a great deal. It has 
provided materials and data I normally could not provide." "The system has 
made it easier to identify learning weeiknesses and remedy them through pre- 
scriptions." "[it offers] .. .more accurate diagnosis and assessment of each 
child's progress for each unit." "[it makes for]. . .better pacing — not going 
beyond class' capabilities nor progressing too slowly." "I know exactly what 
each child is doing and how he compares to his group." "It... helps with 
grouping and regrouping." "I feel more secure when confronting parents during 
conferences." 

However, the teachers did not feel that these benefits were wholly without cost. 
Because it was new to them, and a developmental project, there were trials and 
annoyances. Some teachers were concerned that despite the carrels, children 
were working together (copying) and taUcing while at the earphones. Diffi- 
culties in keeping track of the test each group should be taking were mentioned, 
and some teachers said they were made uneasy by having an observer in the room. ^ \ 
They considered interruptions or visits made without a scheduled appointment an/ 
annoyance. Most teacher suggestions concerned simple mechanics of the syst em / 
such as the need for a better filing system for printouts, the need for ;;iath 
prescriptions, the fact that test numbers should be more prominent and easier 
to find on test booltlets, and the desirability of having wireless headsets for 
presentation of instructions. Some of these suggestions were easy to accommodate 
and were incorporated into the second semester's system. 

VI. CHAHGES I-1A.de TO DiS AS A RESULT OF EVALUATION AITO REVISION 

A. MATERIALS 

Appendix A describes the evaluation and revision procedures used in the initial 
■ development of US tests for the fall 19o7 semester. This process, with its 
many trials and changes to items, was completed before operation of the system 
began in September I967. All tests and materials were located in files in the 
classrooms when school began, to insure that they would be available to the 
teachers at any point in the semester wiien they might be needed. Therefore, 
the design of the tests used in the classes during the first semester was 
"frozen," and major cloanges in the tests were not introduced until the start 
of the second semester. This section of the report gives the rationale for a 
major second semester revision and describes the changes that were made. 

;a 

Pioblems Leading to Revision . Attempts to measure the objectives being taught 
during each week of first-grade reading instruction revealed several problems 
in the design of H»1S tests and pire script ions. First, testing several objectives 
required several formats with several sets of instructions. It was difficult 
to prevent this from becoming so complex tliat the child could not respond 
correctly, even when he knew the answer. Second, dividing items a child could 
co.mplete within 20 minutes ajnong so many objectives meant that most objectives 
had only a few items; accumulating such items over several tests before judging 
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performance against a criterion implied that the child had not changed during 
the time between tests. Third, although prescribing remedial work on the basis 
of nonachievement was a step in the right direction, a single sheet of items 
based on a single criterion level was probably ..too crude a prescriptive process 
to effect many cheuiges in behavior. 

Test Reorganization . To alleviate some of these problems, the con5>lete set of 
reading tests was reorganized. Few items were changed, but many were resequenced 
so that the number of general objectives occurring on any test was reduced by 
half. Rather than having many general objectives appearing in every test, they 
appeared on alternate tests with more comprhensive coverage of specific subject 
matter. To illustrate, instead of having a test of 3U items spread over from 
four to seven general objectives, the revised test would use the same number of 
items to cover only one to three general objectives. Figure 10 shows the item 
breakdown by test and general objective for the revised tests , and provides data 
comparable to that of Figure U for the original tests. 

Advantages to Reorganization . The use of fewer objectives on each test is an 
aid to test administration. Several of the .general objectives are somewhat 
similar in their format of presentation, but require different pupil behaviors. 

In empirical trials during development of the reading tests, students would 
sometimes confuse one set of instructions or format presentation with others, 
resulting in technical errors. While the revised test will not eliminate this 
problem, it should reduce the possibility of such confusions. 

As mentioned before, a second difficulty with the original reading tests was 
that of establishing criterion performance. Even when only three or four items 
were used for the purpose, they had to be accumulated from test to test over 
several weeks* time. This deferment in establishing criterion also meant delay 
in prescribing follow-up work. Once again, the revised tests do not eliminate 
this problem, but they do reduce its frequency of occurrence. To illustrate, 
on the original tests there were 51 items related to the objective of word 
recognition that heeded to be carried forward to establish criterion. The 
revised tests have no instances whatever of items being accumulated before 
criterion is established. Another exsui5>le is the general objective of phonic 
analysis involving initial sounds. Instead of combining items from many 
phonemes on each test, each phoneme is broken out and tested separately. This 
allows prescriptions to be based on single phonemes — that is, on specific 
objectives rather than at the general objective level only. Appendix E shows 
the complete structure for items, by specific objective,- for the revised reading 
tests. 

One of the most critical inprovementS“ln"thB”Te'Vlsed“ test&-jnay have to do with 
the dynamic change in some young children's rate of learning. Suppose a child 
is tested on a specific objective before criterion is established — let us say 
that the tests are given in weeks 1, 3> and 5* The resulting score is based 
on the poor assumption that the child's behavior has not changed between week 
1 and 5> and that no new learning has taken place. Because the revised tests 
do not accumulate items, this problem is eliminated. 
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Problems with Test Reorganization . The reorganization of the reading tests 
required little change in original content but eui almost complete repositioning 
of items within the tests. This is not in every respect an advantage. For 
example, the test revision often narrows the time lapse between test and pre- 
scription, but it Just as often increases the amount of time between instruction 
and test. When the frequency of testing objectives is reduced to alternate 
week, every third week, or every fourth week, the tests often do not coincide 
-with classroom instruction. Thus, some items on a test may have been taught 
several weeks prior to being tested. 

Another problem might arise because of concentration of similar items on each 

test. Some obJectives--notably the phonic analysis euid comprehension behaviors 

gi*/e rise to more complex questions than others; scores are usually lower./ 
Concentration of items on these objectives may prove frustrating to less Advanced 
children. 

Since the advantages of revision seemed to outweigh the disadvantages, the items 
wore repositioned in reading tests for the second semester. Test performance of 
the children, classroom behavior during tests, and teacher opinions concerning 
effects on the changes will be considered in evaluating the revision for the 
second semester. 

B. DATA PROCESSING 

Changes l*xade During First Semester . Three changes or significance were made 
with respect to the data processing system during first semester operations. 

The first was a decision to discontinue recording individual test item responses 
in computer storage. The system was initially programmed to record each student's 
specific choice on each test item. Because of the exorbitant eimount of computer 
time required to insert these data, because the system had no defined use for 
this information, and because the records were available on punched cards, it 
appeared uneconomical to consume computer time for this purpose. It was further 
reasoned tliat if someone (sueh as a test designer) wanted item-specific infor- 
mation, he could perform a special analysis of the data from the records on 
cards. Consequently, program SSU (see Figure 5) was changed. 

The second change was less specific but has rather extensive implications 'for 
the future design of IMS. The initial design philosophy for the system provided 
for an automatically produced, pre structured Progress Report (see Figure 6) that 
would be generated whenever a test'was given to a group of students. In addi- 
tion, teachers were provided with a computer-linked teletypewriter through which 
they could compose an almost unlimited variety of reports as the need for them 
became apparent. In actual practice, hovever, teachers did not use the inter- 
active capability for three reasons: (11 lack of time; (2) lack of familiarity 

with the lan^age to compose reports; an(l (3) the teach^rs' apparent lack of 
interest in identifying situations in which such information might help them. 
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As project members began to realize that the teachers were not initiating 
action to use the available wealth of information about student performance, 
changes to the system were tried. The major change was an effort to prepare a 
computer program that would relieve an interactive user of IMS of much of the 
burden of composing reports. The program, called IMSOL, interactively queries 
the user about his information needs and. con5)ose8 reports automatically. One 
report provides percentage scores for each general objective and each test 
for a designated student, group, or class. A second provides percentage scores 
for the specific objectives under a selected general objective for a student, 
group, or class and for a designated test or test sequence. The third report 
provides a roster--a listing of students' names and numbers--for a designated 
group or class. Examples of these reports are found in Appendix G. Because of 
system problems that occurred in December, IMSOL was not used other than 
experimentally during the first semester. 

A third change in system operations was forced in December when a condition 
occurred that caused the program IMS to become inoperative. When a specific 
attempt was made to operate SSU (see Figure 5), this program was concluded by 
LUCID before it ran to completion. Numerous attempts to continue the program 
failed. The problem was then turned over to SDC's data base people for study 
and eventually two errors in the update function of LUCID were found. In the 
meanwhile, project members ‘turned their attentions to the production of a use- 
ful progress report without using the records stored in LUCID. This was 
accomplished by a listing (produced directly by SSU) that showed each individual 
student's performance on each of the general and specific objectives. Because 
the students' records were not available to SSU, the cumulative scores could not 
be obtained. 

Doth the experience of flying on teachers to define and' obtain information and 
the system difficulties encountered in December have implications for changes 

to IMS during the second semester. These changes are discussed in the following 
section. “ 

Plans for Second Semester. Experience gained during first semester operations, 
enabled, project personnel to define several requirements for the version of D*1S 

that will operate during the second semester. Figure 11 illustrates the second 
semester version. 

System/^f flciency — To reduce the time required to produce the regular 
progress Report, several changes will be made to IMS, The program (SSU) that 
a^lyze^ and scores tests will be combined with the program (PROG) that produces 
report so that the report can be made with minimal interaction with 
j? addition, the single data base used in the first semester version 
into three separate data bases sb that interaction with LUCID 
can be more direct. It is anticipated that these changes will reduce Il-lS 
operator time to one-fourth or less of the tine required in the previous version 
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Definition of Groups - -In the first-semester ^sion, group cumulative scores 
were defined as the average of all scores obtainfed by students who had ever been 
members of a given group (see "Cumulative Score" on Part A, Group Report 
Figure 6 ). Toward the end of the semester, this score contained values carried 
by some stud.ents who were changed to other groups and values of students who had 
left the school. Consequently, this score was very hard to interpret. In ver- 
sion two, the cumulative score will be calculated as the average of nil scores 
carried by present members of a, group. scores 

c. R_e_ports- -Figure 12 shows the format planned for the second-semester version 
of the regular Progress Report. It can be collared with Figure 6 , which is the 
version. The new version will be produced, as before, when a group of 
children take a test. The top two lines contain identification information. 

Jnder these lines, the information is organized first as a group report. A 
group score is provided for each of the general objectives tested and for the 
total test score. As in the first semester version, each general objective 
spore IS a composite of several specifi^c objective scores that are not indi- 
vidually tested. V/henever 80 percent of the pupils , in the group do not score 
dO percent on any of the specific objectives, a code number for a remedial 
activity IS listed under activities recommended. Each code number refers to a 

sheet of practice exercises stored in folders in a filing cabinet located in 
cacn classroom. 

In uie second part of the report, the information is organized by individual 

Btudenfs name-on the same line-is his percentage score 
. thio test. Continuing on the same line are the scores made for each of the 

Snort f ’’y test. Beneeth the naane of cart rtSdent, the 

report Shows those specific objectives on which the student failed to attain 

criterion. The first number identifies the objective. The number in parentheses 
lo the percentage score achieved by the student for the objective. The next 

oSf iJ" follow-up work a^LclSed 

-tudeit ^ is used for each objective on which the 

otudent failed the criterion. 

ThS"'reiL?S'^^ proposed for the second semester.' 

Tais report will be automatically produced by the system each Friday. During 

. irst semester operations, teachers experienced some difficiaty in knowing 

^ivL knowing what test a group should next be 

cLe>’ For TZ7' T . 1^' ‘^^'^sned to assist the teacher in both those 
caseo. For a particular date and subject, the report lists children bv their 

^ child has taken a test by entering his 
percentage score under the test, sequence number. The absence of a score is 

tJr^orf sr;ac"h'cS?r 

onlnoJ? ^ ^ ^re ordered within the groun by their 

umulative scores, .eachers may wish to post this report in the' cl^Issroom. 
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TEACHER 53 
GRADE A1 


DATE 03/19/68 

SUBJECT READING ' 


TEST 9909, GREEN BOOK, PP. 55-63 
GROUP 3 



G;R0UP REPORT 






GENERAL OBJECTIVES 


SCORE 


ACTIVITIES RECOMMENDED 


1 VISUAL DISCRIMINATION 


86 


R9-0109 


7 WORD COMPREHENSION 


69 


R9-0701 R9-0702 


TOTAL SCORE 


80 





INDIVIDUAL REPORT 


0 




STUDENT NAME 


TOTAL 

SCORE 


GENERAL OBJECTIVES^hh^SCORE THIS TEST 
1 7 



GEORGE ANDREWS 




93 


100 


83 


CHARLES MARTIN 




90 


95 


83 


EIAINE' NEWMAN 

702 (50) 


R9-0702 


87 


95 


75 


EVERETT FOSTER 

701 (50) 


R9-0701 


■ 84 


100 


58 


LOUISE WEBER 

109 (71) 

'• 70 (50) 


R9-0109 

R9-0702 


78 


80 


75 


LEE FREDERICKS' 
109 (64) 

701 (62) 

702 (25) 


R9-0109 

R9-0701 

R9-0702 


^3 


'71 


50 


SANDRA WILLIAMS 

106 (71) 

109 (57) 

701 (62) 

702 (50) 


R9-0106 

R9-0109 

R9-0701 

R9-0702 


60 


61 


58 



Figure 12. Proposed Progress Report for Second Semester Version. 




Figure 15* Proposed Classroom Summary Report 
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C. • CIASSROCW PRESENTATION EQUIPMENT 

/ ; 

CbftnpreB Made During the First Semeater . One major change was made to presentation 
eq.ulpment during the first semester* The distractions cauaed by the button-pushing 
feature of the equipment proved to be the biggest annoyance of the system to the 
teacher. By the time that the most advanced classes had completed test 8, it had 
been decided to shift to measured pauses in the place of stops, eliminating the 
need for button-pushing. Taped lessons were recorded continuously, with estimated 
response times allowed between instructions. Some samples were tried out 
successfully, and by the end of November all tapes being recorded were without 
any stops. The teachers enthusiastically accepted the change, since the less- 
complicated system required much less supervision on their part. 

Changes Made for the Second Semester . The decision to remove stops in the tape 
recordings eliminated the need for bulky cable bundles and ready boxes, which 
were not required by a "nonstop" system. Therefore, it was possible to seat 
children more flexibly throughout the room — rigid carrels were no longer required. 
For the spring semester, the carrels are portable . lightweight pieces of chipboard, 
covered with a washable vinyl cloth material and supported by notched wood blocks. 
They can be taken do^m and set up quickly sind easily by students (a very similar 
design has been successfully- used in a kindergarten and first-grade setting for 
about a year), and different positions or configurations may be set iq> readily. 

Speaker wires for the headsets run around the periphery of the classroom, and 
Jacks are provided at various places where the teacher may want to seat groups 
of students. By using three-wire cable, we are able to provide the teacher with 
the ability to have two separate sound sources playing simultaneously, either of 
which may be- selected for any group of students. The teacher will control these 
sound sources (tape or, phonograph) from a table beside her chair. 
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appendix a. 

TRIAL AND REVISION CYCIES IN THE DEVELOPMENT OF IMS TESTS 



Each IMS test was developed In three atepa, an original test construction and 
at least two revisions after the teats were actually administered to sub.lects. 

The original test constnactlon Included specifying objectives and writing Items 
to measure those objectives at the appropriate point in the Instructional 
program. The first revision cycle Included changes to the original test based 
on aralysls of Indlvldua.; trial data. The second revision cycle Included changes 
based on data from group trials of the test. The flow charts of Figures 1, 2, 
and 3 provide an overview of activities Involved In each of the three steps. A 
further discussion of the test development procedures Is presented below. 

A. SPECIFYING OBJECTIVES 



Even when tests are written for existing Instructional programs, the objectives 
^ niust be precisely defined. Several guides were available to 
I^ personnel: publishers' statements, school curriculum guides, and observe - 

instructional activities cf the classroom. Detailed Instructional 
objectives were inferred from these sources for each subject area, and sample 
Items were constructed to Illustrate each objective. At this point teachers 
were consulted to discuss objectives and Illustrative Items. Their suggestions 
for revision guided rewriting of objectives. 

There were two types of objectives: general and specific. The general 

objectives were broadly defined skills associated with successive levels of 

grade. Each general objective was assigned a 
short title, as listed In Table 1. • 

Each general objective Incorporated a number of specific objectives. In some 
cases, these represented specific stimulus conditions under which the behavior 

f exhibited. For Instance, the objective called 

Isual discrimination largely Involved word matching. Sueclflc objectives 
subsumed under visual discrimination Involved matching words when dlstractors 

*he same beglnr.ing letter, etc. In other eases, 
specific objectives essentially represrnted content divisions. For Instance. 

+ ocjectlve of phonic analysis Involved recognition and matching of 

initial and final conaonant sounds in spoken words. The specific objectives 
were a division of this skill by single letter. A listing of all specml 
objective a Is provided In Appendix C. yt^cmc 

B. INITIAL TEST CONSTRUCTION 

The first step In the Initial test-writing cycle was the specification of 
objectives for short units of work. These associated the words or numbers m 
the pupil s most recent lesson with the broad skills or processes descrlbed^n 
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Figure 2 . First Test Revision Cycle 
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Tatlo 1. L'i.st of General Objectives, 

with Ident ifyinr" Code Ilumbers 





Readiness 




Reading 


0100 


Test Orientation Shills 


0100 


Visual Discrimination 


0200 


Visual Matchinc 


0200 


Wora Recognition 


0300 


Letter-Number Recocnltlon 


0300 


Specific Word Confusion 


ouoo 


Word-Phrase Comprehension 


o4oo 


Phonics, Initial Sounds 


0500 


Memory Span 


0500 


Phonics, Final 3ou;;ds 


0600 


Left-Rlsht Progression 


0600 


Phonics, Medial Vowels 


0700 


Phoni c Ana lysl s 


0700 


Structural Analysis 


0300 


Para;;prap'.i Comprehension 


0300 


Word Comprehension 


o;oo 


Ginn Vocabulary ■ 


0900 


Sentence Comprehension 






1000 


Paragraph Compre'henslon 






1100 


Oral Reading 



Arithmetic 

0100 Identifying Sets 
0200 Matchinc Equalities 
0300 Identif:/inG Inequalities 
OlOO Addition 
0500 Subtraction 
0600 Operation Sign 
0700 Order of Numbers 
0300 Groupinc, Place Value 
0900 Planes and Plane Fi(jures 
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the general objectives. Items were then vrrltten and arranged In a scannable 
format for administration. ^ 

There were several restrictions in item writing. An obvious one Is that the 
scanner cannot evaluate constructed response items; therefore, multiple choice 
measurement was used. Another restriction, came from the fact that first-grade 
teachers' using a developmental reading approach do not plan their lessons for 
testing efficiency. They Introduce a small number of vocabulary words each week 
and practice many skills in their context. It is not clear when mastery is 
expected, by all children on particular skills. To test what the teacher was 
doing during any one week required testing many objectives using many item 

formats and narrow content. 

\ 

A third restriction. in item writing involved presentation of tests to first 
graders. Since the pupils cannot read, all directions had to be presented 
orally and many response positions .designated by pictures. Directions were 
presented by tape recording and both oral directions and ambiguity of pictures 
became a part of the test item to be quality-controlled In writing. 

V/orklng within these limitations, project personnel constructed Items for tests, 
one test at a time. Another person on the staff, not Involved in the original 
test writing, reviewed them for clarity and consistency. 

C. FIRST TEST REVISION CYCLE 

As the tests were constructed, each one was taken to a school and tried with 
two children who had completed instruction on skills measured by the test. The 
children were taken from the classroom to a quiet room and the taped instruc- 
tions v;ere played aloud. Any problems during the test were noted by the 
observer. If performance broke down completely, the observer provided any cues 
necessary to get the child to begin again, and noted these. After each child 
completed a test, he was asked to, name the word and picture stimuli on all pages 
and was questioned on any Incorrect responses to provide information to guide 
revision. The notes and tests were analyzed. Revisions were made in the tests 
based upon information obtained from trials with the two children. 

D. TEST REVISION CYCLE 

The revised tests were then taken into the classroom and administered to a group 
of six to ten children at a listening center. As before, an observer noted 
problems during the test. Afterwards, the tests ■v/ere corrected and items 
tabulated and analyzed. If percent or more of the group missed 0 single item, 
stimuli and directions were again examined for possible sources of ambiguity, 
and necessarj' changes were made. Gometlmes the tests went through two cycles 
of group trials before final preparation for printing. 
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E. TYPES OF REVISIONS MADE TO TESTS AND TEST ITEMS 

Many sources of possible ambiguity were discovered in the testing prpcess, and 
when Items v:ere changed to correct these problems, performance usually improved. 

A log of specific changes on each of the 50 tests was maintained. Analysis of 
this log during ongoing test writing revealed that most revisions fell into a 
fairly limited set of categories. Whfen test constructors became alerted to 
these categories, item writing was greatly facilitated. The categories that 
were defined were changes to improve: (l) clarity of single stimuli; (2) the 

relationship of stimuli to others on the page; (3) format; (U) instructions; 

(5) mechanics of presentation; and (6) content to achieve better balance among 
objectives measured. These categories are described below. 

Clarity of Single Stimuli . Many changes were made in'blngle stimuli in order 
to make the specified task of any item as clear as possible to the child. 

Examples of each of these will illustrate the problems involved: 

1. Pictures were changed to avoid ambiguity of labels. For instance, a 
picture of a "Jet" may elicit "aiiT)lane." Candles on a cake may elicit "birthday 
cake" or "candles" instead of just "cake." A picture of an owl may elicit "owl" 
or "bird." An evergreen tree may elicit "Christmas tree." To reduce these 
confusions, efforts were made to simplify pictures or otherwise make their 
identity more obvious. Curtains were removed from stimulus pictures of windcrt^b 
when "v:lndo^^" alone was the desired response. Checks and dots were removed 
from clothing. 



2. Pictures on phonic analysis items were changed to maximize differentiation 
of sound between correct answers and dlstractors. These items required/ the child 
to match a picture to a word from his vocabulary. The child read the /Word, and 
the picture was named. The cue for the correct answer was some sound^in common. 

A typical item looked like this (Figure U): 



P I ay 
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The child was to match the word "play" with the picture of the pipe; the 
Initial "p" sound was the cue to the correct answer. When a dlstractor 
Incorporated any similarity of sound, It tecame a strongly competing stimulus. 

When the cue was beginning sound, rhyming dlstractor s of those with the same 
^inal or medial sounds were avoided. Thus, all of the following examples 
(Figure 5), In which the child Is to match on Initial consonant sound, would be 



fERlC 
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chariKed. The hammer in the top row of Figure 5 would be eliminated because of 
its "r" ending; the rake in the second row would be eliminated .because of the 
medial "a"; and the book in the third row would be eliminated because it rhymes 
with the stimulus • “ ' - 



3- In Hke manner, if the child was matching on final sounds, distractors 
with initial or medial sounds similar to the stimulus word. were avoided. If 
the stimuli were to be matched because they rhymed, acain competing sounds had 
to be eliminated from distractors. Clearly, these items were . only treating the 
first step in development of auditory discrimination. To assess fine discrimina- 
tions, all the examples that were systematically avoided would have to be 
systematically included. 

4. In several instances, figures or pictures were changed to make the 
dimension tested more discriminable . If the child was asked to make a size 
discrimination, the difficulty was reduced by making size differences greater; 
if completeness (or break in figures) was the cue, the difficulty was reduced 
by making the break larger. 

5 . There were changes in figures to eliminate any systematic irrelevant 
cues. This happened particularly in math items, where pattern, picture content, 
or set boundary shape were often cueing choices instead of the number dimension. 
Thus, all of the following examples (Figure 6) would be changed on revision. 

The correct answer in the first item might be cued by both picture content and, 
set boundary shape. The pattern of the rabbits in the second gives it away, 
and the boundary shape in the third provides a cue. In revising the items, a 
careful attempt was made to force attention to the relevant dimension. 

6. Use of stimulus and distractors with common relational concepts was 
avoided. Consider the case where the child is to match a picture of a chicken 
to the word "chair" using the initial "ch" as the cue. A table would not be 
used as a distractor, since its relation to chair is too strong. Again, such 
test items probably should be included later in the sequence, to effect training 
on finer discriminations. 

?• Some pictures were changed to avoid things that are generally 
unfamiliar to six-year olds in the Los Angeles area. For example, these 
children do not often play with sleds, and have no use for mittens. These were 
not used in items. 

3. Finally, test constructors had to guard against their tendency to 
over-use particular stimulus pictures. For instance, a particular fish became 
very popular for representing the "f" sound, as did one snecific .monkey for the 
m sound . 
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Relationships of Stimuli to Others on Page . Items were changed because of the 
ways in vjhich stimuli related to others on a page. For instance, style of 
drawing had to be consistent, and normal proportions had to be maintained; 
darker or heavier lines In one picture or a frog that is several times larger 
than a house in the same Item are both unacceptable. The position of the 
correct response had to be checked and varied to reduce position bias. Items 
I with similar printed stimuli had to be separated by one or more different Items 

, to prevent pupils from losing their places. 

Changes in Format . Changes in format of the material on the page were made 
for reasons of clarity, convenience, and economy. Long* words were shifted to 
ovoid running them together. They were also shifted to avoid their interfering 
. with response spaces or being too close to timing marks or format code (critical 

to the scanner operation) on the page. 



Where the format type called for responses In. a vertical row after comparing 
words or sentences, the response positions were changed from the left to the 
right of the printed stimulus. This allowed the child to read from left to 
right and then mark, thus lessening the possibility of his losing his place. 



r 

ERIC 



The number of items on a page was reduced in. some cases to fit spacing 
requirements of IBM scanner sheets. They were also reduced to make the page 
less complex and easier for the child to respond to, especially in the readiness 
material and early tests in reading. Where only a few pages of a certain type 
or format were being used, they were altered to fit a more basic type format 
to economize on the total number of formats required. 



The order of pages in a test was changed to separate pages of items that were 
alike in format and appearance but that represented different tasks. A page 
for matching initial consonants looks like one where rhyming is desired; when 
two such as these are in sequence, it is confusing to a child who may not be 
listening carefully to the directions. 



Ch anges in Instruction . The need for a change in instructions was recognized 
most often when a number of pupils made the same kinds of errors on a section 
of similar items. Changes were often toward; simpler vocabulary that the 
children could more easily understand. There was a reduction in the amount of 
verbiage to force more responses per instruction (a response for each sentence, 

j 1 R *1 n I o I . ^ 



It was found desirable to add occasional reminders in the instructions to; 
• color only one circle in each row; 
finish each page without help; 
indicate when each page is finished; 

. i put pencils doTO until it is time to respond. 
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M major change was to call out, In the taped Instructions, the names of the 
pictures In phonic Items. Each child was thus exposed to the Interpretation 
tnat was Intended by the experimenters In as unambiguous a form as possible. 
Pauses were Inserted between the Instructions to allow time for marking 
responses. ® 

When new types of tasks In comprehension were Included-in the test, e. page with 
a nonscanned sample item was added to Introduce each new type and lead the 
c-nlld through to the correct response. Because emergencies sometimes arose. It 
was found desirable to Incorporate Into the Instructions the procedure for 
stopping the recorder. 

Changes In Content. The most usual reason for changes In content was to allow 
better balance among objectives In testing. For Instance, visual discrimination 
ioems (a relatively easy objective) appeared early and were dropped after test 
lb. Rhyming was dropped after test I7. A more difficult task, phonic analvsls 
on the basis of ending consonants, was added at test I7. 

In some Instances the time of Introduction of new types of test Items was 
c.nanged to limit the introduction of such new types to one per test. One t\n3e 
of sentence comprehension is Introduced In test k, another In test 7. The tLk 
of seJectlng the logical sentence to follow Is new In test 8, and story 
comprenenslon Is new In test 9. these Instances, there was careful sequenclnp 
01 comprehension Items so as to allow earlier Introduction of those that appeared 



.'.ordlng of paragraphs and answer choices In the paragraph comprehension cages 
was cnanged so that test questions did not require remembering the textbook 
story. The content of the paragraph comprehension items was altered so that 
no paragraphs overlapped In content. The questions themselves were changed In 
wording to correct the grammar, especially where "who" and "whom" were Involved 



^chanlcs of Presentation. Finally, the procedures of presentation underwent 
revision. Early In the group trials, it became apparent that the children were 
sharing InformatJ^on or copying extensively. There was also a great deal of 
otr.er Interaction, such :as~-caiking. To get more reliable results and to reduce 
copying, U-snaped makeshift carrels-we^e constructed from cardboard file divider 
and p^ into service. The most notlceable-dlfference was that the children 
--ootSy^*^^’ cut down greatly and the tests proceeded more 



Otner changes were also made. After working with standard tap\ recorders for 
several months, the staff selected simple cartridge -load recordWs for the fall 
Insta .latlon because they simplified teacher handling. Tests were also 
controlled to ,nalntalri a standard length. All trials were timed ^nd items were 
deleted v/hen any xest took more time to complete than Is available In the 
nor.mal reading period of 20 minutes. 
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Even after these revisions, some test items were still missed "by many children. 

It was difficult to determine whether this was "because the objectives were 
inappropriately stated, "because the items were written poorly, or "because the 
classroom instruction was not teaching the specific skills as intended. The ' 
items were again examined, and the teachers were consulted. If, in th^r 
opinion, the objectives were inappropriately stated, or the items jjoofly written, 
another revision cycle was conducted. Otherwise, the items remained in the test! 
V/hen cycles of revision were completed for each test, .the final form was produced 
on precisely designed IBH scanner sheets, as described in Appendix B. 
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APPENDIX B 

THE DEVEIXDPMINT OF SCANNER PROCEDURES AND FORI-B 



A. SCANNER FORMATS 



Samples of the scanner forms are attached at the end of this appendix. Each 
form is numbered in the lower left-hand corner. Form 0 is the master cover 
sheet. Forms 1-8 show the eight basic test formats designed to accommodate 
the variety of items in the I1<1S tests. 



Specifications for the nine formats were quite rigid. Timing marks, resnonse 
ovals, and format codes had to be precisely aligned, stripes nrinted in reflec- 
ti’/e ink, and sheets consistently of a specified weight and size. These speci- 
fications were submitted to SDC’s printshop and to ^independent printers. The 
lowest bid, one by an outside printer, was accepted. The full order of scanner 
sheets was received from the independent printer three weeks after the final 
proofs for the scanner sheets were submitted. 

Once the basic scanner formats had been accepted, assembling of the field- 
tested H-IS tests into a scanner fonnat was begun. Thirty-line IBM answer 
sheets were marked where the response positions on the corresponding sheet 
formats would be located. Translucent vellum paper was then placed over the 
layout sheets, and the stimuli for the items were arranged and glued into 
position on the vellum. These sheets became the vellum masters from which the 
SDC printers made their plates to print stimuli on test pages. Scanner sheets, 
reflective ink, and vellum masters of the test materials were then delivered to 
the SDC printers for the test printing and assembling operation. As the 
printing of the tests was completed, they were filed in the teachers* filing 
cabinets; then the cabinets were delivered to the classrooms. 



Forms 9 to 12 were designed a little later. They were intended for more 
specialized use. Three of the four were designed to be general purpose answer 
sheets, distinguished from the earlier forms by the absence of space for the 
presentation of stimulus material on the answer sheet itself. The sheets 
provided response positions for 3 -, U-, and 5 -choice multiple choice questions. 
The last format. Form 12, was intended for the casual user (e.g. , a teacher 
other than the four regular participants), to allow the keying of the items on 
individual teacher-made tests to numbered objectives. These forms were also 
ordered from the outside printer, and were delivered in the normal p>-intine 

divided and delivered to the casual- 
user files in the teletype rooms at the schools. 



B. BIS 1232 OPERATION 



As previously mentioned, ’/ariable arrangement of response positions across 

'^®® built-in machine checks on the student mark 

p tions. And. yet, some check had to be made on the quality of responses mad 
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by the first graders. Earlier, it had been decided that the coding of the 
student identification number was too important a responsibility to place on 
first graders. Consequently, a hand preprocessing step was required between 
the time when the students marked the sheet and the time when the sheets were 
scanned. In this preprocessing, trained coders identified each student's name 
from a class roster and coded his number onto the master mark cover sheet. In 
addition, the coder checked the timing mark and format code area for stray 
marks emd checked the response positions for marks that were too short, too 
light, or so long that they would generate more than three punches in a card 
column. Finally, the checker erased marks where the student had crossed out 
one response and chosen another. These procedvires were time consuming but 
were considered necessary to control the quality of the scanner inputs. 

As a further check on the scanning operation, the built-in scanner checks were 
used to verify the legality of the format codes , to check for stray marks in 
the timing mark area, and to check the synchronization between the keypunch and 
the scanner. After scanning, further checks were made on the output deck, 
including a card count and a comparison of the number of cards to the number of 
test sheets. A sight check was also made of the identification punches. For 
each class, student I.D. numbers included the teacher's number coded in the 
first two digits. By holding the deck to the light, the operator could quickly 
check to see that each card.ln-4ihe-deak-for-a gi-ven-tjesting-group had the same 
teacher number. If the decks were punched correctly, light would pass through 
the common punches in the first two columns of the I.D. number. This check 
caught errors both in the coding of the student number and in the operation of 
the master mark featu re . _ ___ _ 

Finally, the decks were interpreted on an laM 057 card -interpreter, and data 
punched from the test input form were verified. Because these data were 
essential in setting the parameters for the processing program, special atten- 
tion was given to checking their validity. These checks performed on the output 
decks^were quite rapid, and could be performed concurrently with the scanning 
of data from another group. After passing these checks, the data were taken to 
an IH-I 3 dO /30 and transferred in binary mode to magnetic tape for final orocessin-'^ 
and input to the system. * 
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APPENDIX C 

READING READINESS OBJECTIVES 



Computer Label 

0100 Test orientation skills 



0101 Mark rows in no. order 



0102 Top, middle, bottom row 



0103 Little -big side of row 



OIOI+ Mark rows top to bottom 



0105 Position cue words 



0200 Visual matching 



0201 Content cue 



Ob.iectlve Statement 

The pupil is able to respond to aural and 
visual cues of row order and stimulus 
position in following directions. 

The pupil is able to select a stimulus 
picture from any numbered row \dien the 
stimulus and the row number are named 
in test directions. 

The pupi.' is able to select a stimulus 
picture from the top row, middle row, or 
bottom row of a three-row page when the 
row stimulus is named in test directions. 

The pupil is able to select a picture on 
a wider side of a row as opposed to a 
narrower side when the picture, row, and 
side of the row are named in test 
directions. 

The pupil is able to select a named 
picture in each row of a six-row pa§p:, 
moving from the top row to the bottom 
' when the directions designate the rows 
as top row. ..next row. ..next row. 

The pupil is able to choose the appro- 
, . priate picture stimulus in a row when 
directions designate it by its position 
relationship (first, last, between, 
right, next to, second, middle, just 
before, left, in front of, the end of) 
to other named stimulus in the row. 

The pupil is able to select from a row 
of visual response choices the one that 
is identical to a printed stimulus. 

The pupil is able to select from a row 
the picture that is identical to a 
stimulus when the choices vary in content. 
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0202 Detail cue 


The pupil is able to select from a row 
the picture that is identical to a 
stimulus \men the choices differ only 
in some detail. 


0203 Direction cue 


The pupil is able to select from a row 
the picture that is identical to a 
stimulus when the choices differ only 
in their spatial orientation. 


0204 Size cue 


The pupil is able to select from a row 
the picture that is identical to a 
stimulus when the choices vary only in 
size. 


0205 Shape cue 


The pupil is able to select from a row 
the abstract figure that is identical 
to a stimulus ^dien the choices are 
different geometric shapes. 


0206 Abstract fig. - detail cue 


The pupil is able to select from a row 
the abstract figure that is identical 
to a stimulus when the choices differ 
only in some detail. 


0207 Abstract fig. - direction 


The pupil IS able to select from a row 
the abstract figure that is identical 
to the stimulus when the choices differ 
only in their spatial orientation. 


0206 Size cue - abstract fig. 


The pupil is able to select from a row 
the abstract figure that is identical 
to tne stimulus when the choices vary 
only in size. 


0209 Numerals 


The pupil is able to select from a row 
the numeral that is identical to the 
stimulus when the choices are numerals 
or letters. 


0210 Lower case letters 


The pupil is able to select from a row 
the lower case letter that is identical 
to the stimulus when the choices are 
letters or numerals of similar shape. 


0211 Upper case letters 


The pupil is able to select from a row 
the upper case letter that is identical 
to tne stimulus when the choices are 
lti>.ters or numerals of similar shape. 
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0212 Upper to lower letters 


The pupil is able to select from a row 
the upper or lower case letter that 
occupies the same alphabetic position 
as a printed stimulus letter of the 
opposite case. 


0213 Two letter words 


The pupil is able to select from a row 
the two-letter word that is identical 
to a printed stimulus word. 


0214 3 + letter words 


The pupil is able to select from a row 
the word that is identical to a printed 
stimulus word when the choices are 
similar words of more than two letters. 


0300 Letter-number recopc. 


The pupil is able to select a named 
letter or numeral from a row of letters 
or numerals. 


0301 Numerals 0-9 


The pupil is able to select a named 
numeral from a row of numerals. 


0302 Capital letters 


The pupil is able to select a named 
capital letxer from a row of capital 
letters. 


0303 Lower case letters 


The pupil is able to select a named * 
lower case letter from a rpw of lower 
case letters. 


0^00 Word -phrase comprehension 


The pupil is able to select a picture 
frojn a row which most appropriately 
illustrates the conditions of a named 
cue. 


OhOl Identify activity 


The pupil is able to select fixim a row 
the picture which most appropriately 
illustrates a named activity. 


0 h 02 Detail of object 


The pupil is ?.ble to select from a row 
of similar pictures the one which 
contains some named detail. 


0U03 Function of object 


The pupil is able to select from the row 
the pictured object which is appropriate 
to some named function (i.e., ... the 
thing you sleep in). 
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0404 Superlative relationship 


The pupil is able to select from a row 
the picture object which appropriately 
illustrates a superlative relationshin 
named in the directions (i.e., the 
biggest dog). 


0405 Negation 


The pupil is able to select from a row 
the picture wiich illustrates a negative 
relationship named in the directions. 


0500 Memory span 


The pupil is able to select three named 
pictures in order, or to select a picture 
that is identical to an earlier presented 
stimulus. 


0501 Recall 3 named objects 


The pupil is able to select a series of 
three pictures, one from each of three 
rows in order as named in the directions. 


0502 Visual i*ecall-directibn 


The pupil is able to select a picture 
that is identical to an earlier presented 
stimulus from a row of pictures differing 
only in direction. 


0503 Abstract direction 


The pupil is able to select an abstract 
iigure that is identical to an earlier 
presented stimulus from a row of figures 
differing only in direction. 


0504 Abstract shape 


The pupil is able to select an abstract 
figure that is identical to an earlier 
presented stimulus from a row of figures 
differing in shape. 


0600 Left-riprht orogression 


The pupil will begin at the left side of 
a row and move right -in naming or drawing 
pictures when directions say to start with 
the "first" or "left" side. 


0601 Draw- sequence as named 


The pupil will begin at the left side of 
a row when asked to copy three named 
pictures. 


0602 First-last as left-right 


The pupil wil], select the left picture 
in a row when asked to choose the "first" 
picture. 
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0603 Left-right of page 


The pupil will select the left picture 
in a row when asHed to choose the ’left" 
picture. 

/ 


0700 Phonic anaJ.ysis 


The pupil will select from a row the 
named picture which has some designated 
sound similarity to a named stimulus 
picture. 


0701 Same initial consoneuit 


The pupil will select from a row the 
named picture which has the same initieil 
consonemt sound as a named stimulus 
picture. 


0702 Rhyming-distractors vary 


The pupil will select from a row of 
named pictures with no sound similarities 
in their labels, the one which rhymes 
with a named stimulus picture. 


0703 Rhyming-vowel sound cue 


The pupil will select from a row of 
named pictures with common final 
consonant sounds in their labels the 
one which rhymes with a named stimulus 
picture. 


O70U Rhyming-consonant cue 


The pupil will select from a row of 
named pictures with common vowel sounds 
in their labels the one which rhymes 
with a named stimulus picture. 


0800 Paragraph comprehension 


The pupil will select from a row the 
picture wiiich .ijost appropriately answers 
questions about a paragraph that has 
been reeid or illustrates a' sequence that 
has been described. 

1 


0801 Sequence of events 


The pupil will select froi.i a row of three 
pictures of temporally related events the 
one which answers the question, "What 
happened first?" or ’’What happened last?" 


0802 Main idea 


The pupil will select from a row the 
picture which most appropriately answers 
a question about the main idea expressed 
in a paragraph that has been read to him. 
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0803 Action seq.uence 


The pupil will select from a row the 
picture which most appropriately answers 
a question about the sequence of events 
in a paragraph that, has been read to 
him. 


O60U Recall details 


The pupil will select from a row the 
picture which answers a question re- 
lating to details of a paragraph that 
has been read to him. 


0805 Drawing conclusions 


The pupil will select from a row the 
picture \diich most appropriately repre- 
sents a conclusion which can be drawn 
from a paragraph that has been read to 
him. 


0900 Ginn vocabulary 


The pupil can match the printed words 
or pictured objects and characters 
taken from given preprimer one to each 
other or to their names aurally presented 


0901 Word recognition 


The pupil can select from a_ row a word 
that has been named in directions. 


0902 Select named ob.jects 


The pupil can select from a row a 
picture that has been named in directions 


0903 Match word to picture 


The pupil can select from a row the 
picture which illustrates a printed 
stimulus word or the word which names 
a stimulus nicture. 
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APPENDIX C 
READING OBJECTIVES 



Computer Label 

0100 Visual Discrimination 

0101 Lower case letters 

0102 Capital letters 



Ob.iective Statement 

The pupil is able to select from a row of 
printed response choices the one that has 
the same cue as a printed stimulus. 

The pupil is able to select from a row of 
lower case letters the one that is identical 
to a printed stimulus letter. 

The pupil is able to select from a row of 
capital letters the one thet is identical 
to a printed stimulus letter. 



0103 Caps to small letters 



0104 Two letter words 



0105 Meiximum difference words 



0106 The same length words 



0107 First letter distractor 



0108 Final letter distractor 



The pupil is able to select from a row of 
capital or lower case letters the one that 
has the same alphabetic position as a 
printed stimulus letter of the opposite case. 

The pupil is able to select from a row of 
two-letter words the one that is identical 
to a printed stimulus word. 

The pupil is able to select from a row of 
words that are different in length and in 
most letters, the one that is identical to 
a printed stimulus word. 

The pupil is able to select from a row of 
words that are alike in length "but different 
in most letters, the one that is identical to 
a printed stimulus word. 

The pupil is able to select from a row of 
printed words with the same initial letters, 
but differing final letters, the one that is 
identical to a printed stimulus word. 

The pupil is able to select from a row of 
words wii:h the same final letters, but differing 
in initial letters, the one that is identical 
to a printed stimulus word. 
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. 0109 Many letter distractors 


The pupil is able to select from a row of 
words that have several letters of '/arying 
positions in common, the one that is 
identical to a printed stimulus word. 


0110 Long words 


The pupil is- able to select - from a row of 
words all more than seven letters long, the 
one that is identical to a printed stimulus 
word. 


0111 Initial letter cues 


The pupil is able to select from a row of 
words that have no particular letter or 
^ngth control, the one which has the sane 
initial letter as a printed stimulus word. 


0112 Caps to non-cap words 


The pupil is able to select from a row of 
capitalized or non-capitalized words with 
no particular letter or length control, the 
same word as a printed stimulus word which 
has an initial letter of the opposite case. 


0113 Word in sentence context 


The pupil is able to select from printed 
words in a sentence one which is identical 
to a printed stimulus word. 


0114 Sentences-beglnning same 


The pupil is able to select from three printed 
sentences that have the sane initial words 
one that is identical to a printed stimulus 
sentence. , 


0115 Sentences -ending same 


The pupil is able to select from three 
^printed sentences that have the same final 
\words one that is identical to a printed 
stimulus sentence. 


0200 Word Hecognition 


The pupil is able to select from, a row of 
printed response words the one that is the 
same as an aural stimulus word. 


0201 Maximum differsnce words 


The pupil is able to select from a row of 
printed words that are different in lengta 
and in most letters, the one that is the same 
as an aural stimulus word. 


0202 Same length words 


The pupil is able to select, from a row of 
printed words with the same number of letters 
tiie one that is the same as an aural stimulus' 

WO Xu « 
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0203 Initial letter distractor The pupil is 'able to select fro.T. a row of 

printed words with initial letters the sane, 
the one that is the saTie as an aural stimulus 
word. 

020i^ Final letter distractor The pupil is able to select from a row of 

printed words with final letters the same, 
the one that is the same as an aural stimulus ■ 
word. 

0205 Many letter distractors The pupil is able to select from a row of 

printed words which have several letters of 
varying positions in common, the one that is 
the same as an aural stimulus word. 

0300 Specific word confusion The pupil is able to select the correct 

printed response word from a row of very 
similar words to match a printed, pictured, 
or aural stimulus.' 



0301 Betty, Bunny 



0302 can, and 



/ 



0303 surprise, something 



O30U airplane, apple 



0305 house, home 



The pupil is able to appropriately select as 
a matching response to a printed, pictured, 
or aural stimulus, the printed word "Betty" 
or "Bunny" from a row of words which includes 
the other. 

The pupil is able to appropriately select as 
a matching response to a printed, pictured, 
or aural stimulus, the printed word "can" 
or "and" from a row of words which includes 
the other. 

The pupil is able to appropriately select as 
a matching response to a printed, pictured, 
or aural stimulus, the printed word "surprise" 
or "something" from a row of words which 
includes the other. 

/ • 

The pupil is 'able to appropriately select as 
a matching response to a prihted, pictixred, 
or aural stimulus, the printed word "airplane" 
or "apple"- from a row of words vmich includes 
the other. 

The pupil is able to appropriately select as 
a matching response to a printed, pictured, 
or aural stimulus, the printed word "house" 
or "home" from, a row of words which includes 
the other. 
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0702 ”s" form of verbs 



0703 "ed" past tense of verbs 



« 



The pupil is able to select from a row of 
printed verbs, some plural, and others 
singular formed by adding an 's,” the one 
vrtiich correctly completes a blank left in a 
printed stimulus sentence. 

The pupil is able to select from three printed 
sentences that are alike except for verb 
tense, the one which most appropriately 
continues the meaning of a printed stimulus 
sentence. 



0800 Word comprehension 



0801 Picture and noun label 



0802 Picture and verb action 



0803 Picture and description 



060h Picture-position word 



The pupil is able to select from a row of 
pictures the one for which a printed stimulus 
label is most appropriate, or to select from 
three sentences the one which continues the 
meaning of a stimulus sentence. 

The pupil is able to select from a row of 
pictures of objects the one which depicts 
the meaning of a printed noun label. 

The pupil is able to select from a row of 
pictures of figures in action the one which 
depicts the meaning of a printed verb stimulus. 

The pupil is able to select from a row of 
pictures illustrating descriptive words the 
one which most appropriately depicts a printed 
stimulus word. 

The pupil is able to select from a row of 
pictures illustrating some position relation- 
ship the one which most appropriately depicts 
a printed stimulus word. 



0805 Picture - "not" phrase 

V 



0806 Use of pronouns 



The pupil is able to select from a row of 
pictures the one which most clearly illustrates, 
a negatively stated stimulus sentence. 



ine pupxj. IS able to select from a row of 
pictures the one which most aoprooriately 
depicts the meaning of a stimulus* oronoun; 
or to select from a list of sentences differi 
only in pronoun the one which correctly refer 
to an antecedent noun in a stimulw^entence. 



\ 
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0807 Color words 



0808 Knowledge of characters 



0900 Sentence comprehension 



. t*- 



0901 Content cues 



0902 Structural cues 



/ 

/ 



0903 Word sequence 

/ 

/ 

/ 

/ 

/ 

1000 Paragraph comprehension 

/ * 

/ 



1001 Sentence relationships 



1002 Main ideas 



The pupil is able to select from a row 01 
pictured objects the one whose t^-pical color 
is the same as a printed color word stLmulus. 

The pupil is able to select from a row of 
pictured characters the one which is named 
by a printed stimulus word. 

The pupil is able bo select from three 
response choices a sentence in which all 
words ai?e in correct sequence and are related 
appropriately in content and structure , or 
select a picture which appropriately illus- 
trates the meaning of a sentence. 

The pi^il is able to select the correctly 
worded sentence from three response choices, 
two of v^'ich have a single word which is not 
appropriate to the content meaning of the 
sent^ce 

Tjhe pupil is able to select the correctly 
^worded sentence from three response choices, 
two of which have a single word which is .not 
appropriate to the content meaning or grammatic 
structure of the sentence. 

The pupil is able to select from three response 
choices the picture which most appropriately 
illustrates the meaning of a printed stimulus 

sentence. 

/ 

The pupil is able to select from three printed 
sentences the one vdiich most appropriately 
continues the meaning of a printed stimulus 
sentence or to select from three pictures the 
one which most appropriately answers a question 
about a printed stimulus paragraph. 

The~papil is able to select from three printed 
sentences the one vdiich most appropriately 
continues the meaning of a printed stimulus 
sentence . 

The pupil is able to select from tiiree pictures 
the one which correctly illustrates the main 
idea of a printed stLmulus paragraph. 



O 
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1003 Seq.uence of action 



1004 Details 



1005 Inferences 



The pupil is able to select front three 
pictures the one which correctly answers 
a question about the sequence of action 
in a printed stimulus paragraph. 

The pupil is able to select from three 
pictures the one which correctly answers 
a question about”sbrae detail of the 
paragraph. 

The pupil is able to select front three 
pictures the one which illustrates the 
most probable inference to be drawn from 
a printed^ stimulus paragraph. 




/ 

\ 

O 
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APPENDK C 

MATHEMATICS OBJECTIVES 



Computer Label 


Objective Statement 


0100 Identifying Sets 


The pupil is able to select from a row the 
set which is identical to a pictured stimulus 
set, or matches an auditory or printed 
numeral description of the set. 


0101 1 to 1 Correspondence 


.The pupil is able to select from a row the 
set which has a number of pictured objects 
equal to the number of objects in a pictured 
stimulus set. 


0102 Empty set 


\ 7 hen directions tell the pupil to 'find the 
empty set," he is able to select from a row 
of sets enclosed in circle boundaries the set 
with no objects. 


0103 Numeral value of sets 


The pupil is able to select from a row the 
set which has a number of pictured objects 
equal to a printed stimulus numeral. 


OlOU Phrase describing sets 


The pupil is able to select from a row ' the 
set which has a number of pictured objects 
■ equal* to * a“ printed numeral~phrase stimulus . 


0105 Set with most members 


When directions tell the pupil to "find the 
. set with the greatest number of things," he 
is able to select from a row the set with 
the greatest number of pictured objects. 


0106 Nonequivalent set 


. When directions tell the pupil to "find the 
set that does not ha'/e the same number of 
things as the other two sets,” he is able to 
select from a row the set with the non- 
equivalent number of pictured objects. 


0107 Set with 1 more, 1 less 


When auditory directions tell the pupil to 
"find the set of (some number) and one more," 
he is able to select from a row the set with 
the number of pictured objects equal to this 
phrase . 


0108 Set with least members 

/ * 


When the pupil is asked to "find the set with 
the least number of things," he is able to 

,select from 'a row the seVwith the least 

number of pictured.^bjects. 
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0?00 Matching Equalities 


The pupil is able to indicate equality between 
numerals, numeral phrases, and words by 
matching those that are equivalent or appro- 
priately selecting the = symbol to express 
equivalent relationships. 


0201 Number, symbols/words 


The pupil is able to select from a row the 
printed numerals or words that are equal to 
each other. 


0202 Phrase/numeral 


The pupil is able to select from a row the 
printed numeral phrases and/or numerals that 
are equal to each other. 


0203 Supplying = 


The pupil is able to select the = symbol when 
it appropriately expresses the relationship 
between two sides of an equation and reject 




> and ,< symbols . 

■ , ■ ' . ^ 


0300 Identifying Inequalities 


The child is able to appropriately identify 
inequality relationships by correctly 
selecting the greatest or least value numeral 
or supplying inequality symbols to express 
nonequlvdlent relationships. 
r. '■ - 


0301 > or < Symbol 


The child is able to select the > or < symbol 
when it appropriately expresses the relation- 
ship between two sides of an equation and 
reject the = symbol. 


0302 Greatest-value numeral 


The child is able, to accurately select from 
a row the greatest numeral when auditory 
directions ask him to do so. 


0303 Lowest-value numeral 


The child is able to accurately ‘select from 
a row the least numeral when auditory 
directions ask him to do so. 


OUOO Addition ■ 

1 


The pupil is able to add tiiree -numeral sams 
and equate two-numeral and three-numeral 
addition phrases. 


0401 Equations, placeholders 


The pupil is jable to select from a row the 
numeral which would accurately fill a 
square placeholder to complete a sum in an 
equation. 
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OUO2 Associative property 



0lf03 Aud. stim-suns to 10 



0404 Vertical sums 



The pupil is able to select from a rov a two 
numeral addition pnrase (i.e., i; + 3) which 
is equivalent to a three-name ral addition 
phrase in a prTntisi stimulus. 

The pupil is able to select from a row the 
sum for an auditorily presented two -numeral 
addition problem. 

The pupil is able to select from a row the 
sum of a column of munerals presented in a 
printed stimulus. 



0500 Subtraction 



The pupil is able to subtract a smaller 
number from a larger within the limits of 
one to ten. 



0501 Equations, placeholders 



G- 



0502 Auditory stimulus 



The pupil is able to select from a row the 
numeral which would accurately fill a 
^uare placeholder to complete a difference 

^/in an equation. 

/ 

0 

The pupil is able to select from a row the 
numeral which represents the difference in 
a horizontally printed stimulus problem. 



0503 Vertical subtraction 



0600 ■ Operation Sign 



OoOl Supply + , - for equation 



OyOO Order of Numbers 



0701 Numeral order 



The pupil is able to select from a row the 
difference between two numerals in a 
vertically printed stimulus problem. 

The pupil is able to select the printed 
addition or subtraction symbol (+ or -) 
"which would correctly fill a placeholder 
to complete a printed stimulus. 

The pupil is able to select the printed 
addition or subtraction sj-mbol (+ or -) 
which would correctly; fill a placeholder 
to complete a pi'inted stimulus equation. 

The pupil can identify numerals missintj 
from a series of pictured objects corres- 
ponding to ordinal words. 

The pupil can select from a rov; ‘tne naneral 
vdiich is missing for a series of printed 
stimulus numerals . 
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0702 Ordinal words, Ist-lOth 


The pupil can select fror* a row the picture.: 
object :.'hich corresponds to an auditorilv 
presented ordinal word. 


0800 Group in.'. Place Value 


The pupil is able to identify the nu;r.ber of 
tens and the number of ones in a two-digit 
numeral. 


0801 Sets of tens and ones 


The pupil is able to select fro;;i a row tiio 
two-digit numeral which corresponds to 
pictured sets of tons and ones. 


0602 Phrase and two-digit no. 


The pupil is able to select from a row the 
two-digit numeral which corresponds to a 
printed word and numeral phrase describing 
its tens place value and ones place value. 


0900 Planes and Plane Figures 

'n. 


The pupil can identify common two-dimensional 
geometric forms and the inside and outside of 
such formfs. 


0901 Identify geometric shapes 


The pupil can select from a row of printed 
responses the common geometric form that is 
named in directions. 


0902 In-, outside geometric 
shape s 


The pupil can select from a row of printed 
responses., the geometric figure which has a 
dot inside or outside its area as nan:ed in 
directions. 
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APPENDIX D 

PIACEMENT OF ITEMS ON FIRST 
SEMESTER TESTS 
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APPENDIX D ■ - 

First Semester Placement of Items by Specific Objecti’/e and Test* 

READING TEST OBJECTIVES 

General Objective j}^l--Vi8ual Discrimination--Match.ing Two Visual Stimuli 
Ob.1. 101 102 /IO3 104 105 106 107 108 109 110 in 112 113 llU 113 Total 



X 

2 

3 

k 

5 

6 

7 

8 

9 

10 

11 

12 

13 

lU 

15 

16 

17 

18 

19 

20 
21 
22 
23' 
2k 

25 

26 
27 



1 2 
k 1 

12 2 

3 11 

2 1 
312 



2 

2 

7 2 
2 
7 



O 

ERIC 



♦General Objectives VI and VII are represented by very few specific objectives: 
therefore no appendix chart is provided. 
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READING TEST OBJECTIVES 
General Objective §11 





Word 


Recognition- 


-Matching a Printed Word to a 


Pronounced Word 


. 4^ 
to 


Maximally 

Different 


Words of 
Same 


Words With 
Initial 


Words With 
Final 


Words With 
Several 




Words 


Lenprth 


Letters Same 


Letters Same 


Letters Same 


] 




2 


1 


2 




— ■ — , 2 
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1 


3 


1 
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2 


4 
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2 


1 


5 






3 
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1 • 
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1 
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7 
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2 


3 


1 / 


8 
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2 


9 






3^ 
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10 
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2 


2 


1 


11 




2 


4 
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12 
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2 


2 , 


13 
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2 


14 






3 


■ 3 




15 




1 


3 


1 . 


2 


16 


/' 


3 


6 


1 


2 


17 




2 


6 


3. 


3 


18 




3 


3 


3 


5 


19 




1 


4 


6 


3 


20 






7 


3 


4 


21 




2 


4 


4 


4 


22 




2 


r 

D 


6 




23 






4 


1 


2 


24 




1 

- 1 


■ 3 


2 


1 


25 




- 


4 


3 




26 




2 


2 


2 


], 


27 






3 


2 


2 



Total 



5 

6 

5 
7 
7 

'7 

7 

6 

7 

. 7 
7 
7 
6 

7 

• 

12 

14 

14 
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14 

14 

14 

7 

7 

7 

7 
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READING TEST OBJECTIVES 

General Objectives j/IV— Phonie Analysis— Determining Initial Consonant Sounds 







I • •• . ' " . 

Figure 3 » Setoiid Test Revision Cycle 




0 



* 



I 



10 May 1968 



■ \. 



^-3298/004/00 



READING TEST OE^CTIVES 
General Objective V--Phonio Analysis --Determining Final Consonemt Sounds 
ObJ. 501 502 503 504 '505 506 507 308 509 510 311 Total 
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. READING TEST OBJECTIVES 
General Objective #IX— Sentence Completion 
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READING TEST OBJECTIVES.. 

1 • ' * 

General Objective^^j^X— Paragraph Comprehension 
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PIACEMENT OF ITEMS ON 
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READING TEST OBJECTIVES * • 

> General Objective IV- -Phonic Analysis — Deteimining Final Consonant Sounds 
Obj. 4 oi 402 4 oj 4 o 4 405 4 o 6 407 4 o 8 - 4 o 9 4l0 411 ^ Tot^l ~ 
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Determining Medial .Consonant Sounds. , 
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• ' General -^Objective Vlr- 

^ . Identifying Pluial Words 

% " ' , ^ (structural Analysis) 
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*Each of these objectives is only represented’ on one test. . 
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READING TEST OBJECTIVES 
General Objective IX--Paragraph Con^rehension 
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IMS'ONf-LINE QUERY PROGRAM - IMSOL 



. ‘ • . -V 

The 3 > 1 S^ On-Line Query program ha^ been designed to allow querying of the D 4 S ’ 
data bases in a manner more oriented to the needs of the II-IS users, rather than' 
..in a general purpose manner such as QUUP .provides. , . 

The following displays are presently available: 

Display i^l. General Objective Scores, provides percentage scores by 
objective and by .total test 'for a student, a group or a class 
Vs ' ■ over several tests with the scores cumulated or listed by test, 

.Display #2, Specific Objective Scores, provides scores for specific . 

' objective^ under a general objective 'for a student, a group or 
a class, over se'veral tests with the scores ctunulated. 

Display y/3 doesn’t exist. k. '■ * 

\ ■ - ' ' ■ 

Display #4, Gioup or Class Roster, provides a list of student numbers 
and student names for a group or a class. 



Following is 'a, guide -to the contents of this document: 

-.j , • 'A- 



■ 4 . .. ■ 



. LOGGING IN AND IPADING . ' ^ 

: ' fDEOTIFYING THE DATA BASE, TEACHER, GRADE, AND SUBJECT 
, .jr-REQUESTING GENERAL OBJECTIVE SCORES 
■"■■REQUESTING SPECIFIC OBJECTIVE SCORES 
/ . .. REQUESTpG A GROUP* OR CIASS ROSTER' 

REQUESTING DISPIAYS FOR ANOTHTO SUBJECT OR TEACHER 
INTERACTING WITH THE TBE-SHARING SYSTEM 
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LOGGING IN AND LOADING. 



/ 



7 



• % 



■ 1 



User: LOGIN XE3^rG12l6 

TSS; $0K LOG ON XX 



User; 

TSS; 



■7, 



r *■ n 



User;--' LOAD IMSOL 
TSS; $LQAD XX 






GO 

$MSG IN. 



0 



IXXjIN Idep^lfles the user to the time- 
sharing system, tSS. The user types 
the .word 'I/XjIN' followed by two numbers] 
the first number is the "man number" or 
"project number" and the second number" •• 
is the "wo’rJt order number" or "share 
nvunber. " ' 

The TSS response includes the' TTY 
channel number on which the user has 
logged.' ' 

r-;-. • (, ■ ■ > 



lOAD requests TSS to', load IMSOL (or any 
user-designated program). .The user 
types the word LOAD' and 'IMOTL. ' If 
the program is not on disc TSS will 
request a tape number ( IMSOL is stored 
on reel number 1920) 

$RPEAT LOAD CMND WITH REEL NUI/iBlSR 
LOAD IMSOL 1920 - . . « 



GO activates IMSOL and transfers control 
from TSS to IMSOL. The user types the- 
word 'GO.' 

H'lSOL is now the' controlling program> 
and to communicate with the user IMSOL 
will print a message on the TTY followed 
by. an asterisk. The. user must wait until 
the asterisk appears before replying. 



.7 



/ 
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ms: 



IDEIITIFYIIIG THE MTA BASE, TEACHER, GRADE, JVND SUBJECT 

"Instructional MANAGEt-osNT system - ii-® 






BIS: IDENTIFY II^PUT DATA BASE. 

User: USE I2/13/67 DISC IMSDA 



f 

• j 


\ 


> 


{ 

h 

. 1 ^ 

• , j ■ 






• i 

■ . j •- 

> 

i 


B'IS: 

User: 


K \ 

. ENTER TEACHER NO. 
■ .*6 


1 

1 

1 

• } 

~ 1 


BIS: 

User: 


ENTER GRADE 
■/ 


i 

••• (i • 

■ 1 


, BIS; 
User: 


ENTER 'SUBJTi'lCT NO. 
*1 


■ ' "! 




^ • > 




■bis.:- 


liOlLDING BRENTWOOD 

-TEACHER' SARAH BURGESS 


ERIC 


■ ’ . ,g 


’ I 



At’ this, point TKQUP, a proceflurized A 
version of QUIP, is read in. If it is \'■^^ 
not found on disc it is read in fror.i 
tape 7^-2053 and the following messages * 
printed: ' \ ' 

■ BIS ROUTINES BEING LOADED FROM TAPE 
.$WAIT \ 

jjjFILE lECQUP DRIVE XX REEt 2053 



The user types the word 'USE' followed ' 
by^the ^name of the data base (the date 
the data base was last, updated). followed 
“by the ^rord 'DISC follovred by the disc - 
name, B'ISDA. ‘ ‘ 

If the^ data base is not found on disc, 
the fbllowing message is .Drinped on the 
TTY: ’ 

DISC ID NOT IN INVENTORY. 

IDEOTIFY INPUT nAilA BASE. • 

The -user would then type the vrar^^ 'USE*' 
followed by the data base name, as ato-,*e, 
folloired by: the word 'TAPE ' followed by 
the tape reel number, ,1007. 

The TSS will resDond vfith: ' •- 

Gwait ' • ' 

,GFILE DBIIIPT DRIVE XX ISEL IOO7 
E ;3 will then respond vfith: 

,100 PERCENT OF iFiPUT ON DISC 



The user types the teacher nif^ber to 
■ wliicn the succeeding display requests 
'will apply. V • ’ 



The user types the grade to which the 
succeeding display re, quests will apply. 



The user types the subject nuTiibef to 
^PPly to the succeeding display requests} 
no. 1 is Reading. 



•DATE 12/20/67 
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IKS; * 

User: 

B43: 



BIS; 

User; 



IMS; 

User; 



B4S; 

User; 



IMS; 



User;- 



IKS; 
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RBftUESTING GENERAL OBJECTIVE SCORES 



ENTER DISPLAY NO. 

1 GENERAL OBJECTIVE SCORES 

2 SPECIFIC OBJECTIVE SCORES 

^3 ■ 

U GROUP OR CIASS ROSTER 



If the, user does not luiow wliat -displays 
are available^ he nay type a question 
mark in response to the request to 
enter a. display no. 



ENTER DISPIAY NO. 
*1 



Th 6 user ‘enters display nOi 1 , General 
' j Objective Scores. 



FOR A STUDENT,. GROUP OR CIASS s7g/c' 
*S ' " . ' . 



ENTER NAME 
*J0HN HAVrrHORNE 



The user enters an - 'S' for a student. 

■ V- 



f 



The user enters the student's name,^ 
first name followed-, by last name. . 



ENTER TEST NOS. 
: *99gil ■ 



'3- 



The user reque^s Gpiieral Objective 
Scores for a qimgle test. 



STUDENT JOHN HAVWHORNE 

SUBJECT B1 READING TESTS 9901-9901 



MTE 12/2 









total ^-’HHHHt-pERCEOT' SCORE BY OBJECTIVE^<‘ X - x x x x 
SCORE 1 2.-3 k . 3 6 7 8 9 . 10 ■ 



?, 



93 - 100 100 71 100 100 



87 . 
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REQUESTmXGENERAL OBJEfcTIVE SCORES (Cont'd) 



mS\ ENTER DISPIAY NO ^ 

User: *1 • , . 

D4S:'» FOR A STUDENT, GROUP OR - CIASS 
User: x-»^S “ 

D'lS:' . '. 'enter NAME 

User: ^JOHN-HAyCTHORNE 

BIS: ■ ENTER TEST NOS. 

User: . *99^1 - 9916 . ' 

B4S : ■ SCORES BY TEST OR CUMUIATED 

User: •itr ^ ^ 



s/g/c ' 



t/c ' . . . ■ • , • . 

'' « 

In -this case, the user is req.uesting the 
General Objective Scores for more than 
one test witSi the scores to be pripted 
out for each test taken. 



BiS: , STUDENT JOHN HAWTHORNE 

SUBJECT B1 READING , TESTS 99 OI -9916 



m 



MTO ia/a^/67 



TEST 

NO 

9901 

9903 

990U 

9905 

9912 

9913 
-9911^ 

9915 

.9916 



TOTAL 

SCORE 

93 

• 89 
87 
67 
9^^ 
-75 

• 75 

.85 

73 



•iHHHHHtpERCENT SCORE BY OBJECTIVE^^'^w*'^ 

1 , 2 3 H 5 6 7 • 8 • 9 10 



100 100 71 100 100 

94 66 100 100 • 

94 100 1.00 83 
88 42 100 ' 50 

100 100 100 
85 100 

83 100' 100 

85 85 100 

58 100 100- 



87 

- 83 

V 83 » y 33 
33 

100 80 

100 75 50 

O 58 100- -57 
' ' 75 80 
70 
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REQUESTING GENERAL OBJECTIVE SCORES (Cont’d). 
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IMS: EI^ DISPIAY NO. 

User: 

H-IS: ^ . TOR A STUDENT, GROUP OR CIASS 

User: *S 

IMS: ENTER miE “ 

User: *J0HN HAWTHOftME 

IMS: ENTER TEST NOS. 

User: *99gil - 9916 

INS: SCORES BY TEST OR CUMUIATED 

User: *c • 



s/g/c 



t/c 

This request is identical to the pre- 
vious request except t^t the scores 
will be cumulated over all tests taken. 



IMS: 



STUDENT JOHN HAWTHORNE 

SUBJECT B1 READING TESTS 9901-9916 



DATE 12/20/67 



O 

ERLC 



total ^HHHH(*PERC&fr SCORE BY OBJECTIVE > ^ x x x x x 

Score 1.2 3 4 ? 6 7 •• 8 9 10 

83 92 '77 86 97 88 87 73 72 68 
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KEXiUESTING GENE31A.L OBJECTIVE SCORES (Cont' 



d) 



IMS: ENTER DISPIAY NO. 

User: *1 



/ 



IMS: FOR A STUBEIW, ^ GROUP OR CIASS S/g/C 
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' IMS: 
User: 
IMS: 
User: 
IMS: 
User: 
IMS: 
User: 
IMS: 
User: 



IMS: 



IMS: 

User: 



ERlCj 
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REQUESTING SPECIFIC OBJECTIVE SCORES 



ENTER 

*2 

FOR k 






DISPIAY NO. 

STUDENT, GROUP OR CIASS 



s/g/c 



ENTER 

*JOHN 



NAME 

HAWTHORNE 



ENTER TEST NOS. ' 

* 99 ^ 

ENTER OBJECTIVE NO. 
*3 



j 



The hser has requested display no* 2, 
Specific Objective Scores, for a student 
for a single test. The specific objec- 
tive requested is no. 3 , Specific Word 
Confusion. 



STUDENT JOHN HAWTHORNE 
SUBJECT B1 READING 



TESTS 9901-9901 



DATE 12/20/67 



OBJECTIVE. 3 SPECIFIC WORD CONFUSION 



^ 'SPECIFIC 
' OBJECTIVE 



BETTY, BUNNY 
CAN, AND 

SURPRISE, SOMtTt 
AIRPIANE, apple/ 
HOUSE, HOME. . / 
WANT, WHAT ' 
SUSAN, SURPRISE 



POSSIBLE STUDENT 
ATTAIN. ATTAIN. PERCENT 



71 



ENTER ANOTHER OBJECTIVE NO. FOR THIS DISPLAY 
*NONE 



The user could have entered another 
objective no., in which case the specific 
objective names and -scores for that 
objective no. would be printed out followed 
by a request for another objective no. for 
this display. When no more specific objec- 
‘ti-ve scores are desired, the user should 
enter 'NONE. ' 



■\ 

- \ 

' \ 
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IMS: 



J 



BIS: 

Usor: 



B:S: 



E-:S: 

User: 



REQUESTING SPECIFIC OBJECTIVE SCORES (Cont'd) 



B-IS: 


ENTER DISPIAY NO. 


User: 


*2 


B<IS: 


FOR A STUDENT, GROUP 


User: 


*s : 


IMS: 


ENTER NAME 


User: 


♦JOHN HA\i/THORNE 


B-IS: 


ENTER TEST NOS. 


User: 


*9901 - 9916 ^ 


IMS:i 


ENTER OBJECTIVE NO. 


User; 


*9 



s/g/c 



•STUDENT JOHIT HAWTHORNE 
SUBJECT B1 READING 




The user has again requested a display 
for Specific Objective Scores for a 
student. This 'time, however, the scores 
will represent the performance over - 
several .tests. 



TESTS 990 ;^ 9916 
OBJECTIVE 9 SENTENCE CO-IPREHENSION 



DATE 12/20/67 



SPECIFIC 

OBJECTIVE 



POSSIBIB STUDENT,' 
ATTAIN. ATTAIN.' PERCENT 



CONTENT CUES 
STRUCTURAL CUES 
WORD SEQUENCE 
PICTURE AND SENTENCE 



12 

6 



11 

2 



91' 

33 



ENTER ANOTHER OBJECTIVE NO. FOR THIS DISPIAY 

*10 



OBJECTIVE 10 PARAGRAPH COMPREHENSION 



SPECIFIC 

OBJECTIVE 



POSSIBLE STUDENT 
ATTAIN. ATTAIN. PERCENT 



SENTENCE REIATIONSHIPS 
NAIN IDEAS 
SEQUENCE OjW^CTION 


20 

k 


Ik 
• k 


70 

100 


DETAILS 


• 15 


9 


60 


INFERENCES 


7 


5 . 


71 



ENTER ANOTHER OBJECTIVE NO. FOR THIS DISPIAY 
*NONE 

After having requested a second set of 
scores for objective 10, the user indi<^ . 

‘ cates that he needs no more data for thii 

• , student over these tests.. \ 






« % 



io May 1968 



12 

» 



0M-3298/oo4/oq 



IMS: 



REQUESTING SPECIFIC OBJECTIVE SCORES (Cont'd) 



s/g/c 



IMS: 


ENTER DlSPIAY NO. 


User: 


*2 


IMS: 


FOR A STUDENT, GROUP OR CLASS 


User: 


■»Kj >-i 


IMS: 


ENTER NAME 


User: 


♦BRONTOSAURUS J 


IMS:, 


ENTm TEST NOS. 


User': 


♦ 991 U 


IMS: . 


ENTER OBJECTIVE NO. 


User: 


*2 



\ • 

\ ■ 
\ 



This request is for Specific*' Objective 
Scores for test 991^ for a grtoup' named 
Brontosaurus, with bhjective 2 being 
requested first. 



GROUP BRONTOSAURUS 
SUBJECT B1 READING ' 



TESTS 991 U- 991 U 



DATE 12 / 27/67 



OBJECTIVE 2 WORD RECOGNITION . 

SPECIFIC ■ 

OBJECTIVE • 



POSSIBLE GROU? 

ATTAIN. ATTAIN. PERCENT 



MAXIMUM DIFFERENCE WORDS 
SAME lENGTH WORDS 
INITIAL lETTER DISTRACTOR 
FINAL LETTER DISTRACTOR 
MANY LETTER DISTRACTORS 



21 

21 



18 

18 



85 

85 



IMS: ENTER ANOTHER OBJECTIVE NO. FOR THIS DISPLAY 

User: *8 ’ -- 



IMS: 



OBJECTIVE 8' WORD CCMPREHENSION 

SPECIFIC 
OBJECTIVE ' 



PICTURE AND NOUN LABEL 
PICTURE AND VERB i\CTION 
PICTURE AND DESCRIPTION 
PICTURE-POSITION WORD 
PICTURE- 'NOT'; PHRASE 
USE "OF/'PRONOUNS 
COLOR WORDS 

KNOWLEDGE OF CHARACTERS 



POSSIBLE GROUP 
ATTAIN. ATTAIN. PERCENT 



7 


7 


100 


lU 


lU 


100 


7 


7 


100 


lU 


14 


100 


42 


40 


1 

95 



IMS: ENTER ANOTHER ;OBJECTIVE NO. FOR THIS DlSPIAY 

User: *NONE 
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. REQUESTING A GROUP OR CIASS ROSTER 

o 

IMS: ENTER DISPIAY NO. 

User: . 

IMS:-^ ' FOR A GROUP OR CIASS G/C 

User: ’■ '' 

IMS: ENTER NAME 

User: *BLUE 

• -- 

The .user has reqiiefted display no. 4, 
a Group or Class Roster, for a group 
* named Blue. (The sample display is for 

teacher no. 55*) .. • • 



D-IS: ■ group’ blue 

SUBJECT B1 READING 



12/20/67 



STUDENT NO. STUDENT NAME"^ 



5 






55^2 


MICHAEL BOVA' 


\ 






55 ^ 


LISA DENBAUGH 


]) . ! - 






5509 . 


GINA GON^IES 








5513 • 


BILLY MCKINNEY 








5516 


BENNY NEEDHAM 


‘ ' ■ ‘V 






5523 


CINDY SZELESTEY 




o 




t 
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REQUESTING DISPIAYS ANOTHER SUBJECT OR TEACHER - . 



IMS: ENTER' DISPIAY NO. 

User: *NONE ' - 






If the user wishes to obtain displays 
about another subject or-another teacher, 
type 'NONE* in response, to the enter 
display no. . . / 



IMS: MORE DISPIAYS FOR TEACHER NO. 6 Y/n 

User: *Y . ' 



In this case, the user wishes additional 
displays about^ teacher no. 6, but wishes 
to change the subject matter. 



IMS: ENTER SUBJECT NO. 

User: *5 



The user requests subject no. 5 , Math. 



IMS: ENTER DISPIAY NO. 

User: * * ' ' 



The user may now continue requesting, 
displays for teacher no. 6 for Math. 






c 



j \ 

j 




j 
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IMS; ‘ ENTER DISPIAY NO. 
User; *NONE 



-Again the user types 'NONE' in order to 
request another teacher no. or subject 
no. 



IMS; MORE DISPIAYS FOR TEACHER NO. 6 y/n 

User; *N 



This time the user wishes to change 
teacher no. 



IMS; OTHER TEACHER'S DISPIAYS Y/n 

User; *Y •• 



If the answer here had been 'N' instead 
of 'Y' the program would have concluded. 
$PGM CONCLUDED 



IMS; 

User; 






TEACHER NO. 



IMS; ENTER GRADE 

User; * 

IMS: ENTER SUBJECT NO. 

User; * 



After the user answers these requests 
the teacher heading' is printed and then 
a display no. is requested, and the 
user is ready to proceed further. 



IMS:. BUILDING BREIfTWOOD 
TEACHER’ 



12/20/67 



t: 



BIS: ENTER DISPIAY NO. 

User: * 



ERIC 
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HfTERACTING WITH THE TIME-SHARING SYSTEM 
A. SPECIAL SYl-lBOIS . . ■ ' 

1 The exclamation point is used to initiate communication with TSS. 

r 

" The quotation mark is used to initiate communication with .IMSOL (or any 
user program that has been Joaded). 

cr The carriage retiufti transmits the TTY teptrt. All inputs must end with a 
carriage return in order for communicatiojjf,to take place ii 



$ The dollar sign precedes 'all messages from TSS. 



/ 



The exclamation point and quotation marks perform another function. If the. 
user wishes to cancel a message he has typed before it has been transmitted, 
i.e., before he hits the carriage return kejc, he may use either the exclama- 
tion points or quotation mark. This will cancel -the' message and Institute a 
carriage return. . ^ • 

B. SPECIAL REQUESTS 

The following requests to TSS can be made without being logged in as a user, 
except for SEARCH. If .the user has been communicating with IMSOL and wishes 
to make a request to TSS, he must precede his request with an exclamation mark. 
Note that once a TSS request is made, it places the user in communication with 
TSS; the quotation mark must be used to ‘ revert to communication with BISOL. 

. i . ' 

.'DIAL , 9 (foUo\red by a message) 

Vfhatever message follows the DIAL 9 request is transmitted to the coniputer 
operator. Each line of the message must be preceded' by the DIAL 9 request. 
TSS Indicates that a line of input has been transmitted by responding with 
$MSG IN. " 



.'DRUMS 



TSS responds with the number of contiguous words of drum space available; 
3^^956 Vros AVBL . \ • 

.'T^ES , /* • 

TSS responds with the number of available tape drives; $3 TAPES'AVIBL 



.'SEARCH FOR (disc inventory name) 

TSS will respond with $N 0 T FOUND or ^ON DISC ONLY, and information about 
the disc file. 

JQUIT ' 

.TSS responds /\-d.th $MSG IN. and 'usually with TTY will be turned off. 

• . . . 



O 
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JSS^TUS 

Following are some of the statuses of the TTY channel the user is usinc- 
$N 0 LOGIN 

The user has not yet logged in. 



I 



$N 0 T LOADED 

The user has not yet loaded IMSOL, 



$VIAITING FOR GO ' • • 

The user has loaded the program, but has not type^'‘'G 0 ' 



$WAITING FOR TAPE 

The^ computer operator has not yet mounted a requested reel. 



$WArrpiG FOR TTY 

The user has not answered the last request by IMSOL or IMSOL did not 
receive the answer. ' Repeat the answer, making sure communication is 
with the program by preceding the answr with the quotation mark. 



PPROGRAM STOPPED ' ' 

Typing *GO*. will again activate^ the program. 



$PGM OPERABLE 



$BTCH QUEUE POSN xx . ' 

The program is waiting for its turn under time sharing. 



$PR 0 D STACK POSN xx 

The program is waiting for its turn under time sharing. 



C. SPECIAL REPLIES 









$n/a 
,:!>what 

JiilLLEGAL Ca#iAND 



TSS does not recognize the response made by the' user. Generally this 
hap^ns user wishes to respond to IMSOL, but has been communi- 

c ting with the TSS and has not preceded the program response with a 
q.uotation mark# 
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D. A SAl-IPIE INTERACTION WITH TSS 



User: 


.'STATUS ■ 


TSS: 


$NO LOGIN 


User: 


LOGIN XE302 GI216 


TSS: 


$ WORK NR. lUEGAL 


User: 

TSS: 


LOGIN XE301 GI216 
?OK LOG ON 11:04.9 I2/27/67 


User: 

TSS, 

!b • 


LOAD IMSOL 

$RPEAT LOAD. CMND WITH REEL NUMBER 


User: 


LOAD IMSOL 1920 i. 


TSS: 


$WAT^' 


TSS: 


$ NO LOAD DRUMS, FULL 


User: 

TSS: 


.'DRUMS ‘ 

^67UWDS AVBL . 


User: 


IDIAL 9 I NEED UTK DRUMS PIEASE • 


TSS: 


$ilSG IN.‘ 


TSS: 


FRCM 0^9 ' JUST A MINUTO PI5. 


TSS:« " 


FROM 09 LOAD NOW 


User: 


LOAD IMSOL ' 


TSS:^ 

1 


$load’xx , 


User: 


22 ' 


TSS: 


\ 

$MSG IN."^ 



9 
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